Speech emotion recognition and model training method, device, equipment and storage medium

Abstract

The present invention provides a method, device, equipment and storage medium for speech emotion recognition and model training, which belongs to the field of speech processing. The model training method provided in the embodiment of the present invention obtains first speech data of a target person, wherein the first speech data includes speech segments of the target person under different emotions; pre-processes the first speech data to obtain second speech data; constructs a first speech emotion recognition model, wherein the parameters in the first speech emotion recognition model include hidden variables y h , the hidden variable y h Including: the speed of speech data #imgabs0# through the second speech data, the first speech emotion recognition model is trained to obtain the second speech emotion recognition model; the above method fully considers the influence of hidden variables on vocal cord movement, can improve the fitting accuracy, thereby realizing the recognition of speech emotions.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a speech emotion recognition and model training method, device, equipment and storage medium. Background Art

[0002] With the rapid development and widespread application of multimedia technology and network technology, the scenarios requiring speech recognition are gradually increasing. At present, speech recognition through machine learning training speech recognition models is widely used.

[0003] However, since speech contains too much information, such as semantics, language, emotions, voiceprints, vocal cord health status, etc., it is difficult for speech-based machine learning to ensure that it learns emotional differences. It is often the case that the results obtained by using existing speech recognition models are quite different from the correct recognition results, resulting in poor speech recognition results. In addition, when there is insufficient speech data, the speech recognition model based on machine learning will have a large deviation. At the same time, in speech emotion recognition, since these methods only process the speech data itself, and emotions are mainly attached to speech through the vocal cords, the accuracy of existing models in emotion recognition is generally low. Summary of the invention

[0004] The present invention proposes a speech emotion recognition and model training method, device, equipment and storage medium, and trains a speech emotion recognition model based on a dynamic model to solve the problem of speech emotion recognition accuracy.

[0005] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:

[0006] In a first aspect, an embodiment of the present invention provides a method for training a speech emotion recognition model, the method comprising:

[0007] Acquire first voice data of a target person, wherein the first voice data includes voice segments of the target person in different emotions;

[0008] Preprocessing the first voice data to obtain second voice data;

[0009] Constructing the first speech emotion recognition model;

[0010] The parameters in the first speech emotion recognition model include hidden variables y h , the hidden variable y h Includes: Voice data speed

[0011] The first speech emotion recognition model is trained using the second speech data to obtain a second speech emotion recognition model.

[0012] Optionally, the preprocessing of the first voice data includes at least one of the following:

[0013] By data cleaning, unclear, semantically unclear and / or distorted data of the first voice data is removed;

[0014] Through semantic content recognition, speech segments containing vowel signals are screened out.

[0015] Optionally, the constructing of the first speech emotion recognition model includes:

[0016] The vocal cord structure is simulated as a vibrating structure of the glottis connected by a spring and a damper. According to the elastic coefficient r of the spring, the damping coefficient c of the damper, the mass m of the glottis, the glottis airflow velocity u, and the glottis air pressure p l And the dynamic relationship between the voice data x, construct the first voice emotion recognition model.

[0017] Optionally, the training the first speech emotion recognition model by using the second speech data includes:

[0018] According to the second speech data, the observed data d is obtained, and the value d(t 0 ) as initial conditions, substituting into the first speech emotion recognition model to obtain initial parameters;

[0019] According to the value of the observed data d at each time, iteratively solve the estimated solution y of the speech data x e , the hidden variable y h and other parameters;

[0020] where t n Indicates the observation time point.

[0021] Optionally, the training of the first speech emotion recognition model by using the second speech data further includes:

[0022] Estimated solution y based on speech data x e The difference between the observation data d and the speech data x constructs y e (t n ) and d(t n ) and optimize the residual function J.

[0023] Optionally, the optimizing the residual function J includes:

[0024] Integrate the elastic coefficient r of the spring, the damping coefficient c of the damper and the glottal air pressure pl into a composite parameter p;

[0025] The residual function J is calculated by the sensitivity vector with respect to the composite parameter The gradient of the sensitivity includes the sensitivity of the composite parameter p, and the sensitivity of the composite parameter p is the partial derivative of the estimated solution with respect to the composite parameter p;

[0026] Update the parameters of the first speech emotion recognition model according to the gradient and the second speech data.

[0027] Optionally, the constructing of the first speech emotion recognition model further includes:

[0028] Divide the observed data d into l segments in time order, and calculate for each segment: the estimated solution, sensitivity, and gradient of the residual function J of the speech data x;

[0029] The sensitivity of a previous segment can affect all segments after the previous segment.

[0030] In a second aspect, an embodiment of the present invention provides a method for speech emotion recognition, comprising:

[0031] Receiving speech to be recognized;

[0032] Obtaining parameters corresponding to the speech to be recognized through a second speech emotion recognition model;

[0033] Determining the emotion reflected by the speech to be recognized according to the parameters corresponding to the speech to be recognized;

[0034] Among them, the second speech emotion recognition model includes the second speech recognition model provided by the first aspect of the embodiment of the present application.

[0035] In a third aspect, an embodiment of the present invention provides a speech emotion recognition model training device, characterized in that the device comprises:

[0036] An acquisition module, used to acquire first voice data of a target person, wherein the first voice data includes voice segments of the target person in different emotions;

[0037] A first processing module, used for preprocessing the first voice data to obtain second voice data;

[0038] The second processing module is used to construct a first speech emotion recognition model, wherein the parameters in the first speech emotion recognition model include a hidden variable y h , the hidden variable y h Includes: Voice data speed

[0039] The third processing module is used to train the first speech emotion recognition model through the second speech data to obtain a second speech emotion recognition model.

[0040] In a fourth aspect, an embodiment of the present invention provides a speech emotion recognition device, comprising:

[0041] A receiving module, used for receiving speech to be recognized;

[0042] A first processing module, used for obtaining parameters corresponding to the speech to be recognized through a second speech emotion recognition model;

[0043] A second processing module, used to determine the emotion reflected by the speech to be recognized according to the parameters corresponding to the speech to be recognized;

[0044] Among them, the second speech emotion recognition model includes the second speech emotion recognition model provided by the first aspect of the embodiment of the present application.

[0045] In a fifth aspect, an embodiment of the present invention provides an electronic device comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the speech emotion recognition model training method of the first aspect are implemented; or, the steps of the speech emotion recognition method of the second aspect are implemented.

[0046] In the sixth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the speech emotion recognition model training method of the first aspect are implemented; or, the steps of the speech emotion recognition method of the second aspect are implemented.

[0047] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0048] The speech emotion recognition model training method provided by the present invention obtains first speech data of a target person, wherein the first speech data includes speech segments of the target person under different emotions; preprocesses the first speech data to obtain second speech data; constructs a first speech emotion recognition model, wherein the parameters in the first speech emotion recognition model include hidden variables y h , the hidden variable y h Includes: Voice data speed The first speech emotion recognition model is trained through the second speech data to obtain the second speech emotion recognition model; the above-mentioned speech emotion recognition model training method fully considers the influence of hidden variables on vocal cord movement, improves the fitting accuracy, and trains the model through the second speech data. It is possible to determine the model parameter distribution corresponding to speech with different emotions, thereby realizing the recognition of speech emotions. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0050] Figure 1 A flow chart of a speech emotion recognition model training method provided by an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of the steps for optimizing a speech emotion recognition model provided by an embodiment of the present invention;

[0052] Figure 3 A flow chart of a method for speech emotion recognition provided by an embodiment of the present invention;

[0053] Figure 4 A schematic diagram of the structure of a speech emotion recognition model training device provided by an embodiment of the present invention;

[0054] Figure 5 A schematic diagram of the structure of a speech emotion recognition device provided by an embodiment of the present invention;

[0055] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Currently, when performing speech emotion recognition, the traditional approach is to use sentiment classifiers for recognition. The so-called sentiment classifiers can be divided into two general types: linear classifiers and non-linear classifiers.

[0058] Among them, linear classifiers can be further divided into two types: based on the discriminative characteristics of the applied speech signal itself and through the probability density function. The basic idea of ​​the linear classifier is to derive the emotion label through the weighted linear combination of the speech signal, and use methods such as linear regression to train the weight vector so that the new speech signal can be input into the trained system to label it with an emotion; or, use the method in the nonlinear classifier.

[0059] For example, Support Vector Machine (SVM) is the most commonly used nonlinear classifier, another popular random method is Hidden Markov Model (HMM), Gaussian Mixed Model (GMM) is a probabilistic method different from the first two methods, Artificial Neural Network (ANN) is one of the highlights of advanced computing research and has also been applied to emotion recognition.

[0060] All of the methods described above are random or statistical methods. However, since speech contains too much information, such as semantics, language, emotions, voiceprints, and vocal cord health status, in the absence of sufficient speech data, it is difficult for the machine learning-based speech recognition training model to ensure that the learned speech emotion differences are the same. This application solves the above problem and improves the accuracy and precision of the speech emotion recognition model through the following method.

[0061] Please refer to Figure 1 , Figure 1 A flow chart of a speech emotion recognition model training method provided by an embodiment of the present invention is shown, and the method comprises:

[0062] Step 11, obtaining first voice data of a target person, wherein the first voice data includes voice segments of the target person under different emotions.

[0063] Specifically, step 11 is used to collect voice data for training the model, including data collected from mobile phones, computers and other devices, and data downloaded from public data sets on the Internet, in formats such as WAVE, MPEG, MP3, WMA, etc.

[0064] It is worth noting that the speech data collected in this method is not the entire speech signal, but rather speech fragments containing different emotions of the target person.

[0065] Step 12: preprocess the first voice data to obtain second voice data.

[0066] Specifically, the preprocessing operation deletes the speech data that affects the fitting accuracy in the first speech data, and selects the second speech data for training the first speech emotion recognition model.

[0067] Step 13, constructing a first speech emotion recognition model, wherein the parameters in the first speech emotion recognition model include a hidden variable y h , the hidden variable y h Includes: Voice data speed

[0068] Due to the complexity of glottal oscillation, when we build the model, we need to consider that some variables are difficult to observe during model training, such as the speed of vocal cord movement and other influencing factors. When we build the model, this part of data that is difficult to observe is called hidden variables. Specifically, when building the model, in one embodiment provided by this application, the speed of speech data is used to To represent the hidden variable y h .

[0069] Specifically, the speech data x here can represent the scale of vocal cord movement, the speed of speech data The corresponding physical meaning can represent the speed of vocal cord movement.

[0070] Step 14: train the first speech emotion recognition model using the second speech data to obtain a second speech emotion recognition model.

[0071] The speech emotion recognition model training method provided in the embodiment of the present application fully considers the impact of hidden variables that cannot be observed during vocal cord movement, performs modeling, and improves the fitting accuracy of model parameters, thereby determining the parameter distribution of speech with different emotions in the first and / or second speech emotion recognition model, improving the accuracy and precision of speech emotion recognition, and can solve the problem that when speech data is insufficient, the speech recognition model based on machine learning is difficult to ensure that what is learned is the emotion distinction.

[0072] Optionally, the preprocessing of the first voice data includes at least one of the following:

[0073] By data cleaning, unclear, semantically unclear and / or distorted data of the first voice data is removed;

[0074] Through semantic content recognition, speech segments containing vowel signals are screened out.

[0075] Specifically, only speech segments containing vowel data are used for training, which improves the fitting accuracy of model parameters.

[0076] Specifically, since most of the consonant pronunciation vocal cords do not vibrate when people speak, if the vocal cord model is used to analyze the consonants, the results will be greatly affected. Therefore, the vowel segments of the same person with different emotions and the same words are used for analysis.

[0077] Exemplarily, vowel signals can be screened out by controlling the semantic content and performing semantic content recognition, and then only the speech segments containing vowel signals are analyzed and the model is trained, which can improve the fitting accuracy of the model parameters.

[0078] Exemplarily, the preprocessing also includes data cleaning, by which unclear, unclear, distorted and other data existing in the voice data collected from the device are removed, a high-definition data set is retained, and the training efficiency and accuracy of the model are improved.

[0079] Optionally, the constructing of the first speech emotion recognition model includes:

[0080] According to the vocal cord structure, the first speech emotion recognition model is constructed: the vocal cord structure is simulated as a vibrating structure of the glottis connected by a spring and a damper. According to the elastic coefficient r of the spring, the damping coefficient c of the damper, the mass m of the glottis, the glottis airflow velocity u, and the glottis air pressure p l And the dynamic relationship between the voice data x, construct the first voice emotion recognition model.

[0081] The speech emotion recognition model training method provided in the embodiment of the present application constructs a first speech emotion recognition model by analyzing the vocal cord structure, so that the obtained model can be more consistent with the dynamic principles of human vocalization, and obtain more accurate speech emotion recognition model training and recognition effects.

[0082] According to the structure of the human larynx, the vocal cords extend over the glottis, which separates the trachea from the upper part of the larynx, also known as the pharynx. Both vocal cords are attached to the tubercle of the lung. The epiglottis, with its anterior end located below the epiglottis and its posterior end connected to the cricoid cartilage by a ring, allows the vocal cords to abduct (move away from the inside of the body) and adduct (move toward the inside) In addition, the vocal cords on both sides are connected to the tracheal wall on the same side, with the folds of the ventricles, which are stretched when the vocal cords adduct, ensuring that only fine grooves, when the glottis is closed, allow air to be expelled from the trachea.

[0083] Due to the high complexity of the glottal oscillation process, the present application creatively establishes a so-called lumped model to describe the main dynamic characteristics of the vocal cords. The present application simulates the vocal cord structure as an oscillating structure connected to the glottis by a spring and a damper, and in some cases, the air flow velocity of the glottis is coupled with the subglottal and supraglottal pressures through the Bernoulli equation. In principle, the lumped mass model converts the complex physical problem of human voice oscillation into a single variable or multivariable dynamic problem. The lumped model is simple and efficient. The most basic lumped vocal cord folding model is a mass and one degree of freedom system.

[0084] Changes in voice intonation can reflect the speaker's emotions. For example, the volume of speech will increase unconsciously when one is emotionally excited. The volume is directly related to the vibration amplitude of the vocal cords. For example, when one is emotionally excited, one will unconsciously raise the pitch. The vibration of the vocal cords is a very complex structure, which is at least related to the airflow provided by the lungs and the structure of the vocal cords themselves. The pulmonary airflow provides the vibration force for the vocal cords to vibrate, causing the vocal cords to vibrate. The pulmonary airflow includes the pulmonary air pressure and airflow velocity. The structure of the vocal cords themselves includes the mass of the vocal cords and the elasticity of the vocal cords. Therefore, in the model, springs and dampers are used to simulate the elasticity of the vocal cords.

[0085] For example, the following uses a basic one-mass and one-degree-of-freedom model to simulate the vibration of the vocal cords. The vocal cord structure is simulated as a vibrating structure of the glottis connected by a spring and a damper. According to the elastic coefficient r of the spring, the damping coefficient c of the damper, the mass m of the glottis, the glottis airflow velocity u, and the glottis air pressure p l And the dynamic relationship between the voice data x, construct the first voice emotion recognition model, and obtain the ordinary differential equation:

[0086]

[0087] Here x represents the speech data, m represents the quality of the glottis, u represents the glottal airflow velocity, and p represents the l glottis (lung) pressure, r spring elasticity parameter, c damper parameter.

[0088] We use this model to fit real speech data, and equation (1) can also be modified as follows:

[0089]

[0090] The speech data x is the data that can be observed. The speed of the speech data is a hidden variable in this system, which is a variable that cannot be observed, p l It is the glottal air pressure that is the parameter that needs to be fitted;

[0091] r is the elastic parameter of the spring, and c is the damping coefficient of the damper. These two parameters are usually known parameters and are used together with the glottal air pressure p in the subsequent model optimization. l Together they form a composite parameter.

[0092] Optionally, the training the first speech emotion recognition model by using the second speech data includes:

[0093] According to the second speech data, the observed data d is obtained, and the value d(t 0 ) as initial conditions, substituting into the first speech emotion recognition model to obtain initial parameters;

[0094] According to the value d(t n ), iteratively solve the estimated solution y of the speech data x e (t n ), the hidden variable y h and other parameters;

[0095] where t n Indicates the observation time point.

[0096] Specifically, equation (2) clearly shows that the system actually has three independent parameters, namely the elastic parameter r, the damping coefficient c and the glottal pressure p l , we can solve the ordinary differential equation (2) to obtain the estimated solution y representing the vocal cord vibration scale (i.e., speech data x) at each observation time point e (t n ).

[0097] Specifically, regarding the emergence of hidden variables, ideally, data of all components of the state variables in the model are available. However, in the observed data, we often only observe part of the state variables, which leads to the emergence of hidden variables.

[0098] Optionally, the training of the first speech emotion recognition model by using the second speech data further includes:

[0099] Estimated solution y based on speech data x e The difference between the observation data d and the speech data x constructs y e (t n ) and d(t n ) and optimize the residual function J.

[0100] Specifically, the observation time point t n When the estimated solution y e (t n ) and the observed data d(t n ) There must be a difference. When this difference is smaller, it means that the parameters in the model are closer to the actual situation, so it is necessary to minimize this difference. e (t n ) and observed data d(t n ) to construct the residual function J:

[0101]

[0102] in, is the estimated solution y of equation (2) e In t n The state vector at time, is the observed data d at t n The state vectors at each moment are converted into vector form for the convenience of fitting calculation.

[0103] Optionally, the optimizing the residual function J includes:

[0104] The elastic coefficient r of the spring, the damping coefficient c of the damper and the glottal air pressure p l Integrate into a composite parameter p;

[0105] The residual function J is calculated by the sensitivity vector with respect to the composite parameter The gradient of the sensitivity includes the sensitivity of the composite parameter p, and the sensitivity of the composite parameter p is the partial derivative of the estimated solution with respect to the composite parameter p;

[0106] Update the parameters of the first speech emotion recognition model according to the gradient and the second speech data.

[0107] Specifically, the process of optimizing and calculating the residual function J is as follows:

[0108] Differentiating according to the parameters of equation (2) yields the gradient of the residual function J (in Einstein notation):

[0109]

[0110] p is the parameter damping coefficient, elastic coefficient r and glottal pressure p l The subscripts i and j are the tensor subscripts in Einstein notation.

[0111] Specifically, Represents the residual function J relative to the composite parameter The gradient of y j (t n ) is the time t n Estimate the solution y e (t n ) Einstein said that d j (t n ) is the time t n Observation data d(t n ) shows that we can estimate the solution only if we know the partial derivatives of the solution with respect to the parameters The gradient of the residual function J can be calculated only when the value of is the sensitivity of the system. Therefore, we now need to solve the sensitivity problem of the state variables.

[0112] There are three general types of sensitivity calculation methods: (1) finite difference schemes, (2) internal difference methods, and (3) methods that solve sensitivity equations. The first two methods have the disadvantage of adjusting certain parameters or integrating models for different problems, while the latter method is invariant for different problems. To derive the sensitivity equations, we differentiate the original differential equation (2) with respect to the parameters. For general differential equations: in is the composite parameter vector, we get (in Einstein notation):

[0113]

[0114]

[0115] Among them, y i ,y j ,y k To estimate the solution y e The Einstein representation of the subscript k is the tensor of the Einstein representation, f i is the Einstein representation of f, and equation (5) is the obtained sensitivity equation.

[0116] By taking the initial conditions Solving the derived sensitivity equation (5) with the original differential equation (2), we can obtain the estimated solution y i Its sensitivity To solve the sensitivity equation, we sometimes first solve the original equation (2) of the estimated solution, and then use it as a known constant in the sensitivity equation to calculate the sensitivity of the state variable p to different parameters, and then calculate the gradient of the residual function J with respect to the parameter p. Finally, by letting the update parameter use:

[0117]

[0118] Update formula (7) to update the parameters.

[0119] When solving differential equations, the embodiments of the present application can efficiently and accurately solve the above ordinary differential equations by improving the multiple targeting algorithm. The improved multiple targeting algorithm is referred to as a variational multiple targeting algorithm.

[0120] Specifically, due to the huge fluctuations of real speech signals, such as volume, pitch, speech speed and other influencing factors, the traditional differential equation calculation method may not be able to provide an accurate and effective solution. In addition, the original multiple targeting algorithm cannot solve the hidden variable problem in the speech emotion recognition model proposed in this application.

[0121] Therefore, we designed a variant version of Bock's multi-targeting algorithm based on the gradient descent method to update parameters. This algorithm is more compatible with the gradient descent method and the characteristics of the hidden variable problem in the speech emotion recognition model given in the embodiments of the present application.

[0122] First, as with the traditional multi-targeting algorithm, the time domain is divided into several time periods, such as dividing a 10s speech data into 2-2-2-2-2 seconds. The residual function J is kept unchanged as the estimated solution y in the entire time domain. e The sum of squares of the differences between the estimated value of the observable variable and the observed data d.

[0123] It is worth noting that here, all the segment initial conditions of all variables in each segment are not considered as unknown parameters, but only those segment initial conditions with hidden variables are considered as unknown parameters. (The segment initial conditions here specifically refer to the parameters of each segment at the initial moment. For example, a 10s speech segment is divided into 5 2-second segments, and the c1 segment is the 0-2s segment. The initial condition of c1 is at t=0, according to the observed data d(t 0 ) obtained in the model; c2 segment is the 2nd to 4th second, and the initial condition of c2 is when t = 2. According to the observed data d(t 2 ) The parameters in the model obtained by ; and so on. ) can be implemented by defining a new composite parameter vector, which is a combination of all the parameters in the model, and obtaining the gradient of the residual function J, the parameters in the model, and the initial condition estimates of the hidden variables in each segment by calculating the sensitivity of the state variables with respect to the model.

[0124] The multi-targeting algorithm needs to know the observed values ​​of all variable data at some intermediate time points. However, in the hidden variable problem, the observed data of the hidden variables at the intermediate time points cannot be missing, only the data of the observable variables. Therefore, it is necessary to take the observed data of the hidden variables at the intermediate time points as parameters and optimize them together with other parameters in the model. By updating the parameters to reduce the residuals, find the appropriate generalized model parameters (including other parameters and the intermediate observed values ​​of the hidden variables) to make the residuals as small as possible. While parameterizing the initial conditions of the hidden variables, taking the observed data of the hidden variables at the intermediate time points as parameters and optimizing the parameters of the model at the same time can solve the problem of hidden variables in the model and improve the fitting accuracy.

[0125] Finally, in the next iteration, the estimated solution is obtained by solving the model differential equations in each segment separately. The initial conditions of each segment model are given by: The estimated solution is directly given by the observed data d (the initial time of each segment ), and the initial conditions of the implicit variables (this part cannot be observed, the initial conditions of each segment )composition.

[0126] refer to Figure 2 A schematic diagram of the process of optimizing the speech emotion recognition model proposed in the embodiment of the present application using a variational multi-targeting algorithm is shown.

[0127] Optionally, the constructing of the first speech emotion recognition model further includes:

[0128] Divide the observed data d into l segments in time order, and calculate for each segment: the estimated solution, sensitivity, and gradient of the residual function J of the speech data x;

[0129] The sensitivity of a previous segment can affect all segments after the previous segment.

[0130] It is worth noting that the sensitivity formula for the composite parameter p is: In a composite parameter vector, there are two types of parameters, one is p in the above formula l , c, r; the other type is the initial conditions of the hidden variables in each segment. For explicit variables, that is, observable variables, when solving differential equations on each segment, the values ​​of the explicit variables observed at the beginning of each segment can be used as initial conditions. However, since the values ​​of hidden variables cannot be observed, the initial values ​​of hidden variables in each segment must be fitted at the same time. The method of fitting all these parameters requires solving the sensitivity equations simultaneously.

[0131] Among them, p l , c, r are all global parameters, that is, p l The values ​​of , c, and r have the same effect on each segment, such as the first segment or the fourth segment, because p l , c, and r are inherent parameters of the mechanical model.

[0132] However, the initial conditions of the hidden variables of each segment are local parameters, that is, the initial conditions of the hidden variables of the first segment will affect all subsequent segments, but the initial conditions of the hidden variables of the second segment can only affect the subsequent segments and not the first segment.

[0133] Therefore, there are actually two types of sensitivity equations. The first is for p l The sensitivity equations of , c, and r are to be solved in all segments. The second type is the sensitivity equations of the initial conditions of the hidden variables of different segments. For example, the sensitivity equations of the initial conditions of the hidden variables of the second segment are invalid in the time interval of the first segment, and are only valid in the segment time interval from the second segment onwards.

[0134] Secondly, the initial condition of the sensitivity equation of the model parameters is zero. In the initial condition of the sensitivity equation, both the explicit variable and the hidden variable are derived from the initial condition of the hidden variable in each segment as the initial condition. In the initial condition of each segment, the initial value of the sensitivity of the hidden variable in each segment is 1, and the initial value of the sensitivity of the explicit variable in each segment is 0. For p l , c, k parameters, the initial sensitivity condition is also 0. The initial value of the hidden variable in each segment is the initial sensitivity of the hidden variable itself relative to its initial value in each segment is 1. The initial sensitivity of the explicit variable in each segment relative to the initial value of the hidden variable is also 0. Because the initial conditions of the variables and hidden variables in each segment are irrelevant, the derivative of the explicit variable with respect to the initial conditions of the hidden variable in each segment is 0. At the same time, the gradient formula of the residual function J in this case is different. The partial derivative of the residual function J with respect to the composite parameter p will be calculated in the entire time domain, while the partial derivative of the residual function J with respect to the piecewise initial conditions of the hidden variables in each segment is only calculated in the local segment.

[0135] Exemplarily, the training method of the speech emotion recognition model is further described below in combination with the variational multi-targeting algorithm.

[0136]

[0137] 1. Segmentation based on time domain. According to the second speech data, the observation data d is segmented into l segments in time sequence, and is expressed as the initial condition of the lth segment, Including the initial conditions of observable variables and hidden variables. By splitting the time domain to train the model, the initial conditions of the hidden variables can be connected in series, making it easier to expose the hidden variables.

[0138] 2. Set hidden variables. A vector representing the estimated solutions for the observable variables, represents a hidden variable vector, which cannot be observed.

[0139] 3. Define the residual function J. The residual function is defined as

[0140] 4. Define the composite parameter p. denotes the vector of initial conditions for the hidden variables in segment l, and denotes the vector of composite parameters as The composite parameter vector can then be is defined as the concatenation of these initial conditions for the model parameter vector and the hidden variables, where l represents the number of segments:

[0141]

[0142] 5. Find the estimated solution y e The estimated solution for each segment is computed using the initial conditions given by the observed data d and the initial conditions of the hidden variables obtained from the estimated composite parameters in the last iteration step.

[0143] 6. Sensitivity equation. Derive the composite parameters about the entire time domain And the initial conditions of the hidden variables for the lth segment The sensitivity equation is shown in the equation:

[0144]

[0145] 7. Sensitivity calculation. Use the estimated solution Solve the sensitivity equations. For the sensitivity of their variables to the jth model parameter, the initial condition is: Regarding the sensitivity of the i-th variable of the l-th segment to the j-th initial condition of the segment, the initial condition is:

[0146]

[0147] Where e(j) represents the estimated variable vector Hidden variables in The index of the jth component of l is the time of the starting point of segment l.

[0148] 8. Gradient calculation. Based on the sensitivity of the estimated solution to the composite parameter, the gradient of the residual function is calculated:

[0149]

[0150]

[0151] Among them, s l The data point index representing the starting point of the lth segment in the data sequence in the entire time domain, T l represents the number of data points in segment l, y ej (t n ) represents the estimated solution (including x and estimated solution).

[0152] 9. Update parameters. Update the composite parameter vector using the update formula:

[0153] Finally, the parameter distribution in the model corresponding to different voice emotions can be determined, and the second voice emotion recognition model can be obtained. The actual voice data (such as 10086 manual customer service recordings) is fitted to judge the voice emotions.

[0154] refer to Figure 3 , an embodiment of the present invention provides a method for speech emotion recognition, comprising:

[0155] Step 31, receiving the speech to be recognized;

[0156] Step 32, for obtaining parameters corresponding to the speech to be recognized through a second speech emotion recognition model;

[0157] Step 33, for determining the emotion reflected by the speech to be recognized according to the parameters corresponding to the speech to be recognized;

[0158] Among them, the second speech emotion recognition model includes the second speech emotion recognition model provided in the embodiment of the present application.

[0159] In summary, the speech emotion recognition model training method and speech emotion recognition method provided in the embodiments of the present application have the following effects:

[0160] 1) In the data preprocessing process of this method, in the selection of speech data, vowel segments of the same person with different emotions and the same word are used for analysis, and the vowel signals are selected for analysis to improve the fitting accuracy.

[0161] 2) This method makes full use of the dynamic principle of vocal cord sound production, innovates based on the traditional multiple-targeting algorithm, and solves the problem of hidden variables in the model. Through the second speech data obtained after preprocessing, the parameters are estimated iteratively to determine the parameter distribution of speech with different emotions in the model, thereby realizing automatic speech emotion recognition.

[0162] Moreover, in the application field, the technical solution provided by the present invention can solve the problem that users cannot understand their emotions during mobile communication. It can provide concrete reference and help for customer service quality inspection, make the evaluation results more objective, and ultimately help enterprises improve customer service quality, improve customer experience, and increase customer satisfaction.

[0163] refer to Figure 4 , an embodiment of the present invention provides a speech emotion recognition model training device 40, comprising:

[0164] An acquisition module 41 is used to acquire first voice data of a target person, wherein the first voice data includes voice segments of the target person in different emotions;

[0165] A first processing module 42, configured to pre-process the first voice data to obtain second voice data;

[0166] The second processing module 43 is used to construct a first speech emotion recognition model, wherein the parameters in the first speech emotion recognition model include a hidden variable y h, the hidden variable y h Includes: Voice data speed

[0167] The third processing module 44 is used to train the first speech emotion recognition model through the second speech data to obtain a second speech emotion recognition model.

[0168] Optionally, the preprocessing of the first voice data includes at least one of the following:

[0169] By data cleaning, unclear, semantically unclear and / or distorted data of the first voice data is removed;

[0170] Through semantic content recognition, speech segments containing vowel signals are screened out.

[0171] Optionally, the second processing module 43 is further used for:

[0172] According to the vocal cord structure, the first speech emotion recognition model is constructed: the vocal cord structure is simulated as a vibrating structure of the glottis connected by a spring and a damper. According to the elastic coefficient r of the spring, the damping coefficient c of the damper, the mass m of the glottis, the glottis airflow velocity u, and the glottis air pressure p l And the dynamic relationship between the voice data x, construct the first voice emotion recognition model.

[0173] Optionally, the third processing module 44 is further used for:

[0174] According to the second speech data, the observed data d is obtained, and the value d(t 0 ) as initial conditions, substituting into the first speech emotion recognition model to obtain initial parameters;

[0175] According to the value of the observed data d at each time, iteratively solve the estimated solution y of the speech data x e , the hidden variable y h and other parameters;

[0176] where t n Indicates the observation time point.

[0177] Optionally, the third processing module 44 is further used for:

[0178] Estimated solution y based on speech data x e The difference between the observation data d and the speech data x constructs y e (t n ) and d(t n ) and optimize the residual function J.

[0179] Optionally, the third processing module 44 is further used for optimizing the residual function J, including:

[0180] Integrate the elastic coefficient r of the spring, the damping coefficient c of the damper and the glottal air pressure pl into a composite parameter p;

[0181] The residual function J is calculated by the sensitivity vector with respect to the composite parameter The gradient of the sensitivity includes the sensitivity of the composite parameter p, and the sensitivity of the composite parameter p is the partial derivative of the estimated solution with respect to the composite parameter p;

[0182] Update the parameters of the first speech emotion recognition model according to the gradient and the second speech data.

[0183] Optionally, when constructing the first speech emotion recognition model, the second processing module 43 is further used to:

[0184] Divide the observed data d into l segments in time order, and calculate for each segment: the estimated solution, sensitivity, and gradient of the residual function J of the speech data x;

[0185] The sensitivity of a previous segment can affect all segments after the previous segment.

[0186] The beneficial technical effects of a speech emotion recognition model training device 40 provided in an embodiment of the present invention are the same as those of a speech emotion recognition model training method provided in an embodiment of the present invention, and will not be repeated here.

[0187] refer to Figure 5 , an embodiment of the present invention provides a speech emotion recognition device 50, comprising:

[0188] A receiving module 51, used for receiving speech to be recognized;

[0189] A first processing module 52 is used to obtain parameters corresponding to the speech to be recognized through a second speech emotion recognition model;

[0190] A second processing module 53 is used to determine the emotion reflected by the speech to be recognized according to the parameters corresponding to the speech to be recognized;

[0191] Among them, the second speech emotion recognition model includes the second speech recognition model provided in the embodiment of the present application.

[0192] The beneficial technical effects of a speech emotion recognition device 50 provided by the embodiment of the present invention are the same as those of the speech emotion recognition method provided by the embodiment of the present invention, and will not be repeated here.

[0193] Please refer to Figure 6The embodiment of the present invention further provides an electronic device 60, including a processor 61, a memory 62, and a computer program stored in the memory 62 and executable on the processor 61. When the computer program is executed by the processor 61, each process of the above-mentioned speech emotion recognition method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0194] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned speech emotion recognition method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0195] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0196] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or an electronic device, etc.) to execute the methods described in each embodiment of the present invention.

[0197] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A speech emotion recognition model training method, characterized in that: The method comprises: Acquire first voice data of a target person, wherein the first voice data includes voice segments of the target person in different emotions; Preprocessing the first voice data to obtain second voice data; Constructing the first speech emotion recognition model; The parameters in the first speech emotion recognition model include hidden variables y h , the hidden variable y h Includes: Voice data speed Using the second voice data, the first voice emotion recognition model is trained to obtain a second voice emotion recognition model; The step of constructing a first speech emotion recognition model comprises: According to the vocal cord structure, the first speech emotion recognition model is constructed: the vocal cord structure is simulated as a vibrating structure of the glottis connected by a spring and a damper. According to the elastic coefficient r of the spring, the damping coefficient c of the damper, the mass m of the glottis, the glottis airflow velocity u, and the glottis air pressure p l and the dynamic relationship between the speech data x, constructing a first speech emotion recognition model; The first speech emotion recognition model is constructed by the following formula:

2. The speech emotion recognition model training method according to claim 1, characterized in that: The preprocessing of the first voice data comprises at least one of the following: By data cleaning, unclear, semantically unclear and / or distorted data of the first voice data is removed; Through semantic content recognition, speech segments containing vowel signals are screened out.

3. The speech emotion recognition model training method according to claim 1, characterized in that: The step of training the first speech emotion recognition model by using the second speech data includes: According to the second speech data, the observation data d is obtained, and the value d(t0) of the observation data d at time 0 in the time domain is used as an initial condition, and substituted into the first speech emotion recognition model to obtain initial parameters; According to the value of the observed data d at each time, iteratively solve the estimated solution y of the speech data x e , the hidden variable y h and other parameters; where t n Indicates the observation time point.

4. The speech emotion recognition model training method according to claim 3, characterized in that: The step of training the first speech emotion recognition model by using the second speech data further includes: Estimated solution y based on speech data x e The difference between the observation data d and the speech data x constructs y e (t n ) and d(t n ) and optimize the residual function J.

5. The speech emotion recognition model training method according to claim 4, characterized in that: The optimizing the residual function J comprises: Integrate the elastic coefficient r of the spring, the damping coefficient c of the damper and the glottal air pressure pl into a composite parameter p; The residual function J is calculated by the sensitivity vector with respect to the composite parameter The gradient of the sensitivity includes the sensitivity of the composite parameter p, and the sensitivity of the composite parameter p is the partial derivative of the estimated solution with respect to the composite parameter p; Update the parameters of the first speech emotion recognition model according to the gradient and the second speech data.

6. The speech emotion recognition model training method according to any one of claims 2 to 5, characterized in that: The constructing of the first speech emotion recognition model further includes: Divide the observed data d into l segments in time order, and calculate for each segment: the estimated solution, sensitivity, and gradient of the residual function J of the speech data x; The sensitivity of a previous segment can affect all segments after the previous segment.

7. A method for speech emotion recognition, characterized in that: The method comprises: Receiving speech to be recognized; Obtaining parameters corresponding to the speech to be recognized through a second speech emotion recognition model; Determining the emotion reflected by the speech to be recognized according to the parameters corresponding to the speech to be recognized; Wherein, the second speech emotion recognition model includes the second speech recognition model as described in any one of claims 1-6.

8. A speech emotion recognition model training device, characterized in that: The device comprises: An acquisition module, used to acquire first voice data of a target person, wherein the first voice data includes voice segments of the target person in different emotions; A first processing module, used for preprocessing the first voice data to obtain second voice data; The second processing module is used to construct a first speech emotion recognition model, wherein the parameters in the first speech emotion recognition model include a hidden variable y h , the hidden variable y h Includes: Voice data speed A third processing module, configured to train the first speech emotion recognition model using the second speech data to obtain a second speech emotion recognition model; The step of constructing the first voice emotion recognition model comprises: constructing the first voice emotion recognition model according to the vocal cord structure: simulating the vocal cord structure as a vibration structure in which a glottis is connected by a spring and a damper, and constructing the first voice emotion recognition model according to the dynamic relationship between the elastic coefficient r of the spring, the damping coefficient c of the damper, the mass m of the glottis, the glottis airflow velocity u, the glottis air pressure pl and the voice data x; The first speech emotion recognition model is constructed by the following formula:

9. A speech emotion recognition device, characterized in that: The device comprises: A receiving module, used for receiving speech to be recognized; A first processing module, used for obtaining parameters corresponding to the speech to be recognized through a second speech emotion recognition model; A second processing module, used to determine the emotion reflected by the speech to be recognized according to the parameters corresponding to the speech to be recognized; Wherein, the second speech emotion recognition model includes the second speech recognition model as described in any one of claims 1-6.

10. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the speech emotion recognition model training method as described in any one of claims 1 to 6 are implemented; or, the steps of the speech emotion recognition method as described in claim 7 are implemented.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the steps of the speech emotion recognition model training method as described in any one of claims 1 to 6; or, implements the steps of the speech emotion recognition method as described in claim 7.

Images