Cross-speaker small sample speech emotion recognition method based on meta-learning

Abstract

The invention discloses a cross-speaker small sample speech emotion recognition method based on meta-learning, and belongs to the fields of speech signal processing, emotion recognition and machine learning. According to the method, a mechanism of learning in meta-learning is utilized, meta-model parameters with strong generalization ability are learned through collaborative optimization of multiple meta-tasks in a training stage, so that when a meta-model faces a new task, only a small number of samples with emotion labels are required to be finely adjusted, and emotion recognition of each voice can be realized; and the accuracy of speech emotion recognition under the small sample condition is improved. Furthermore, when the meta-task is generated, the speaker and the emotion are subjected to binary grouping to obtain the meta-category, and the identity information of the speaker and the emotion feature are deeply coupled, so that the model can learn the specific emotion expression mode of the speaker and the cross-speaker common emotion feature at the same time, and the cross-speaker speech emotion recognition accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of speech signal processing, emotion recognition and machine learning, and particularly relates to a cross-speaker speech emotion recognition method. BACKGROUND

[0002] Speech emotion recognition, that is, recognizing the emotional state of the speaker in real time from the speech signal, such as happy, sad, angry, surprised, disgusted and frightened.

[0003] In recent years, speech emotion recognition plays an increasingly important role in intelligent customer service, mental health monitoring, human-computer interaction and other fields. In the field of intelligent customer service, a speech emotion recognition system can optimize service strategies through real-time emotion analysis (for example, recognizing when to transfer to a human customer service when the user is dissatisfied); in the field of mental health monitoring, a speech emotion recognition system can detect emotional tendencies such as depression and anxiety from speech to assist remote psychological assessment; in the field of human-computer interaction, a speech emotion recognition system can endow virtual assistants and social robots with more natural emotional feedback capabilities. Traditional speech emotion recognition methods generally train models through speech data with emotion category labels, and then perform emotion recognition through the trained models. Although speech emotion recognition has a wide range of application scenarios, traditional emotion recognition methods have some problems: 1. Emotion annotation data is scarce, and the annotation cost is high due to the dependence on professional knowledge and subjective ambiguity (such as the boundary between 'neutral' and 'calm'); traditional speech emotion recognition learning methods require a large amount of labeled data, making it difficult to adapt to small sample scenarios with scarce labeled data; 2. The acoustic performance of the same emotion under different ages, genders and accents is significantly different, and the cross-speaker generalization ability of existing models is often insufficient; the accuracy of emotion recognition varies greatly with different speakers.

[0004] Meta-learning (Meta-Learning) can quickly adapt to new tasks from a small number of samples through the mechanism of "learning to learn". The most representative example of meta-learning is Model-Agnostic Meta-Learning (MAML), which is a general meta-learning framework. The core idea of MAML is to optimize the initial parameters of the model so that the model can quickly adapt to new tasks with only a small number of samples and a few gradient updates. MAML performs excellently in small sample learning tasks, but requires the calculation of second-order gradients, resulting in large memory and computational resource consumption. The emergence of the Reptile algorithm simplifies the calculation, requiring only first-order gradients, and through the method of aggregating parameters after multiple steps of stochastic gradient descent iterations, it avoids the calculation of second-order derivatives and speeds up the training of the model. Meta-learning provides a solution to the problems of data scarcity and high annotation cost in speech emotion recognition.

[0005] Chinese patent 202211010973.4 relates to an adversarial and meta-learning method based on a speaker emotional speech synthesis model, which mainly optimizes the emotional speech synthesis system using adversarial training and meta-learning. It includes data preprocessing, design of an end-to-end speech synthesis base model, and adding an adversarial training module to erase the timbre information in the emotional embedding representation. The adversarial module mainly includes a gradient reversal layer and a speaker classifier and meta-learning-based training. The meta-learning method is used to improve the generalization ability of the emotional speech synthesis model, enabling it to quickly adapt to a small amount of speaker voice data.

[0006] Chinese patent 202011536734.3 relates to a few-shot image emotion classification method based on meta-learning. First, a plurality of meta-learning tasks similar to the few-shot images with emotional label information in the target dataset are constructed on the source dataset. Then, through learning these meta-learning tasks, a good classification model initialization parameter is obtained, so that the classification model can achieve good classification effect when facing emotional images in the few-shot target dataset. This technical solution not only greatly alleviates the need for labeled data, but also conforms to the human learning method (humans learn new tasks based on tasks they have learned) based on meta-learning, making the neural network model more intelligent. SUMMARY

[0007] To solve the problems of lack of emotional annotation data, high annotation cost, and the need for large amounts of training data for traditional speech emotion recognition learning methods, which makes it difficult to adapt to small sample scenarios, and the problem of insufficient cross-speaker generalization ability due to significant acoustic differences in the same emotion under different ages, genders, and accents, a cross-speaker few-shot speech emotion recognition method based on meta-learning is proposed. Existing methods mostly use independent modeling of speaker features and emotional features, ignoring the interaction between the two. This invention constructs a (speaker, emotion) binary combination class as a meta-class, maps the emotional features of different speakers to a shared feature representation space, and simultaneously identifies the speaker and emotional label. Through the construction of meta-tasks, the speaker identity information and emotional features are deeply coupled using meta-learning, enabling the model to simultaneously learn speaker-specific emotional expression patterns and cross-speaker common emotional features, improving cross-speaker speech emotion recognition accuracy. This invention uses meta-learning to learn the learning mechanism, and through multiple meta-task cooperative optimization in the training phase, learns meta-model parameters with strong generalization ability, enabling the meta-model to quickly adapt to new tasks. Only a small amount of emotional labeled samples are needed to fine-tune the meta-model, which can realize the emotion recognition of each voice and improve the accuracy of speech emotion recognition under small sample conditions.

[0008] This invention proposes a cross-speaker few-sample speech emotion recognition method based on meta-learning, as shown in the appendix. Figure 1 As shown, the process includes six steps: speech feature extraction, meta-category generation, meta-model construction, meta-model parameter initialization, outer loop update, and target task adaptation. Each step is as follows:

[0009] Step 1) Speech Feature Extraction: The emotion2vec model is used for speech emotion feature extraction. A 768-dimensional speech emotion feature vector X is extracted from each input speech using the emotion2vec model. Emotion2vec is a commonly used speech emotion feature extraction model. This invention uses the IEMCOAP dataset as the training dataset, which is widely used for speech emotion recognition.

[0010] Step 2) Meta-category generation: A dual grouping mechanism based on sentiment features and speaker identity is adopted. By combining sentiment features with speaker information, a binary combination category (speaker, sentiment) is constructed to achieve fine-grained meta-learning task partitioning. Specifically, it includes the following steps:

[0011] Step 2.1) Define the speaker set S = {s1, s2, ..., s...} i , ..., s I}, where s i Let I represent the i-th speaker, and I be the total number of speakers; the sentiment tag set E = {e1, e2, ..., e...} j , ..., e M}, where M represents the number of sentiment tag categories, e j This represents a specific type of sentiment label. Based on the speaker identity label and the sentiment category label, all samples in the training dataset are jointly grouped according to the binary combination of (speaker, sentiment). Each group constitutes a meta-category, i.e., the meta-category c is defined. ij =(s i e j ), where s i ∈S, e j ∈E, which means taking the j-th emotion of the i-th person as the meta-category c. ij Step 2.2) Repeat the grouping method in Step 2.1) to obtain the set of all meta-categories. And the corresponding meta-category labels, the total number of meta-categories is I×M.

[0012] Step 3) Meta-model construction: The 768-dimensional speech emotion feature vector X from Step 1) is input into a fully connected neural network with two layers. The first layer is a pre-feature compression layer, consisting of a fully connected layer with an input dimension of 768 and a ReLU activation function, compressing the input features from 768 dimensions to F dimensions, where F ranges from 200 to 350, and outputting F-dimensional features h. The second layer is a classification output layer, consisting of a fully connected layer and a softmax activation function, mapping the F-dimensional features to an M-dimensional meta-class space and outputting the probability distribution of each meta-class.

[0013] P(y|X)=Softmax(W out h+b out )

[0014] Where P(y|X) represents the probability of outputting label y given the input speech emotion feature vector X, and W out It is an M×F dimensional weight matrix, b out For bias. Final recognition result y * That is, the label y with the highest probability. * =argmaxP(y|X)

[0015] Step 4) Metamodel parameter initialization: Initialize the metamodel constructed in step 3) using f θ Let θ represent the meta-model parameters, and let θ be randomly initialized.

[0016] Step 5) Outer Loop Update: The meta-model parameters are updated through the outer loop, with each outer loop cycle updating the parameters once. This includes five steps: generating the meta-task, inner loop update, calculating the learning rate, updating the meta-model parameters, and determining whether to stop. Each step is explained in detail below:

[0017] Step 5.1) Generate meta-tasks: Using a random sampling strategy, generate meta-tasks from the meta-category set C obtained in Step 2); set the meta-task complexity to m, that is, each meta-task contains m meta-categories, and the meta-task complexity should be consistent with the final number of sentiment categories to be identified; randomly select M meta-categories from C to form one meta-task T. i For each selected meta-category, all samples of that meta-category are extracted from the speech emotion feature vector set as the training set for that meta-task. Simultaneously, the meta-task T... i All samples were randomly shuffled; as shown in the attached document. Figure 2 As shown, a total of K meta-tasks are generated, and each meta-task has M categories, denoted by T1, T2, ..., T... KThis approach uses the binary combination of (speaker, emotion) as a meta-category and forms a meta-task, enabling the model to learn both speaker and emotion information simultaneously when processing the meta-task. It focuses on the intrinsic connections and differences between speaker identity and emotion information, strips away speaker features, focuses on emotion features, and improves the accuracy of cross-speaker emotion recognition.

[0018] Step 5.2) Inner loop update: as shown in the appendix Figure 3 As shown, the process includes five steps: extracting the meta-task, copying the meta-parameters, updating the parameters, calculating the parameter difference, and determining whether to terminate. A detailed explanation of each step is as follows:

[0019] Step 5.2.1) Extract meta-tasks: Extract the meta-tasks T1, T2, ..., T generated in Step 5.1) sequentially according to the number of iterations of the inner loop. i , ...T K The i-th inner loop extracts the meta-task T i As a training set.

[0020] Step 5.2.2) Copy meta-parameters: Copy the current meta-model parameter θ to the current task-specific meta-model: φ i ←θ,θ i For the current meta-task T i The corresponding meta-model parameters yield the meta-model specific to the current meta-task.

[0021] Step 5.2.3) Parameter Update: Update the parameters for the current meta-task T. i The cross-entropy loss function is calculated using the following formula:

[0022]

[0023] Where J is the current metatask T i The number of samples, X j It is the current task T i The speech emotion feature vector of the j-th sample in the dataset. It is X j Input into the current task-specific metamodel The output of the m-th node in the subsequent output layer. j，m For the corresponding classification label, if the j-th sample belongs to the m-th class, then y j，m =1, otherwise y j，m =0.

[0024] Perform backpropagation to calculate gradient Update the parameter φ of the current meta-task-specific meta-model. i for

[0025] Where β is the learning rate of the inner loop. In this embodiment, β = 0.01.

[0026] Step 5.2.4) Calculate the parameter difference: Calculate the parameter φ of the updated current meta-task-specific meta-model. i The difference Δθ between the current metamodel parameter θ and the current metamodel parameter θ i :

[0027] Δθ i =φ i -θ

[0028] Step 5.2.5) Determine if the loop has ended: Check if the loop has run K times. If yes, exit the inner loop and execute step 5.3). If no, repeat steps 5.2.1) to 5.2.5) until the loop has run K times. Extract K meta-tasks T1, T2, ..., T through the inner loop. K And perform an inner loop update for each meta-task, calculating the loss. and the corresponding parameter differences Δθ1, Δθ2, ..., Δθ K .

[0029] Step 5.3) Calculate the learning rate: Update the learning rate α of the outer loop using a cosine annealing learning strategy, with the following formula:

[0030]

[0031] Where α0 is the initial value, t is the number of outer loop iterations (if it is the nth outer loop, then t = n), and N is the total annealing cycle, which is the total number of outer loop iterations. α will dynamically change as the number of outer loop iterations t increases.

[0032] In this example, the initial value α0 = 0.1, and the total number of outer loop iterations N = 150.

[0033] Step 5.4) Metamodel parameter update: Based on the parameter differences Δθ1, Δθ2, ..., Δθ obtained in step 5.2), K The learning rate α obtained from (and 5.3) is used to update the meta-model parameters.

[0034]

[0035] Step 5.5) Determine whether to stop: Based on the loss obtained in step 5.2). Calculate the change in average loss of the outer circulation.

[0036]

[0037] Determine if the average loss change |ΔL| over W consecutive outer loops is less than the threshold δ. If so, stop the outer loop and set θ. (t) As the optimal meta-model parameter θ *; if no, judge whether the outer loop has been cycled N times, if yes, stop the outer loop, and set the meta-model parameter θ (t) as the optimal meta-model parameter θ * , if no, set θ (t) as the meta-model parameter θ for the next outer loop, i.e. θ (t) , continue to repeat steps 5.1)-5.5). As shown in the attached Figure 4 , the meta-model parameter is updated every time the outer loop is passed.

[0038] Step 6) Target task adaptation: The target task is to perform speech emotion recognition, which can identify E = {e1, e2,..., e j ,..., e M} M types of emotions. It includes three steps of training data preparation, target task specific meta-model fine-tuning, and target task recognition.

[0039] Step 6.1) Training data preparation: Use speech data with E = {e1, e2,..., e j ,..., e M} M emotion labels as target task training data. Extract 768-dimensional speech emotion feature vectors for each speech according to step 1). Since meta-learning has the ability to learn to learn, the number of classes in the target task needs to be consistent with the number of meta-classes in step 5.1), but the specific emotion categories in the target task can be different from the emotions in step 2).

[0040] Step 6.2) Target task specific meta-model fine-tuning: Load the optimal meta-model parameter θ * , and input the target task training data into the optimal meta-model to calculate the cross-entropy loss as

[0041]

[0042] where T is the number of target task training data, X t is the speech emotion feature vector of the tth sample in the target task training data, is the output of the mth node in the output layer of the optimal meta-model after inputting X t , y t,m is the corresponding classification label, if the tth sample belongs to the mth class, y t，m = 1, otherwise y t，m = 0.

[0043] Perform backpropagation to calculate the gradient , fine-tune the optimal meta-model, and obtain the parameter θ # of the target task specific meta-model as

[0044]

[0045] Step 6.3) target task recognition: extract the 768-dimensional speech emotion feature vector from the speech to be recognized according to step 1), and pass the speech emotion feature vector through the target task-specific meta-model to obtain the probability corresponding to M types of emotions, and the emotion with the maximum probability is the result of emotion recognition, and the speech emotion recognition is completed. Compared with the prior art, the present application has the following beneficial effects:

[0046] The present application proposes a cross-speaker small sample speech emotion recognition method based on meta-learning, which has significant advantages in solving the key problems of small sample speech emotion recognition practical application, mainly in the following aspects:

[0047] 1) effectively solve the small sample learning bottleneck: traditional deep learning models for speech emotion recognition usually need a large amount of labeled data for training, which faces the challenges of high labeling cost and data scarcity in reality. The present application innovatively introduces a meta-learning framework, especially a unique (speaker, emotion) binary combination meta-class construction strategy in the "meta-task generation" phase. This strategy simulates the process of "learning how to learn", and through training on a large number of and diverse meta-tasks (each task corresponds to a different combination of speaker and emotion), the model quickly masters the core ability to learn new emotion recognition tasks from small sample data. This makes the model achieve excellent recognition performance even if only a small amount of labeled samples are needed for fine-tuning when facing the target task, greatly reducing the model's dependence on large-scale labeled data.

[0048] 2) good cross-speaker generalization ability: the great differences in intonation, accent and pronunciation habits of different speakers are the main interference source of emotion recognition, which seriously affects the generalization effect of traditional models on new speakers. The "binary combination mechanism" and "meta-class generation" in the present application force the model to pay attention to the internal correlation and difference between speaker identity and emotion information when processing meta-tasks. By constantly identifying the same emotion of different speakers and the different emotions of the same speaker in the training, the model learns to strip the individual characteristics of the speaker and focus on the core expression features of the emotion. The rapid adaptation process to different (speaker, emotion) combination tasks in the inner loop update further strengthens the cross-speaker robustness. Therefore, the model trained by meta-learning can more effectively transfer emotion knowledge to speakers who have never been seen before, significantly improving the recognition accuracy and stability in cross-speaker scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is the overall block diagram of the method of the present application.

[0050] Figure 2 is a schematic diagram of the meta task result generated by the method of the present application.

[0051] Figure 3 is a flowchart of the inner loop update in the method of the present application.

[0052] Figure 4 is a schematic diagram of the meta model parameter update by the outer loop in the method of the present application.

[0053] Figure 5 is a flowchart of the target task adaptation in the method of the present application. DETAILED DESCRIPTION

[0054] The present application proposes a cross-speaker small sample speech emotion recognition method based on meta learning, characterized in that, as shown in the accompanying Figure 1 illustration, it includes six steps of speech feature extraction, meta class generation, meta model construction, meta model parameter initialization, outer loop update, and target task adaptation. Each step will be further described in detail below in combination with the accompanying drawings and specific embodiments:

[0055] Step 1) Speech feature extraction: emotion2vec model is used for speech emotion feature extraction. The model is an open source model, and the corresponding open source code address is https: / / github.com / ddlBoJack / emotion2vec . The present application uses the emotion2vec model to extract a 768-dimensional speech emotion feature vector X from each input speech. Emotion2vec is a commonly used speech emotion feature extraction model. The present application uses the IEMCOAP dataset as the training dataset, and the IEMCOAP dataset is a widely used dataset for speech emotion recognition.

[0056] Step 2) Meta class generation: a dual grouping mechanism based on emotion features and speaker identity is used to jointly construct (speaker, emotion) binary combination classes by combining emotion features and speaker information, and to realize fine-grained meta learning task division. Specifically, the following steps are included:

[0057] Step 2.1) Set the speaker set S = {s1, s2,... s i ,..., s I}, where s i represents the i-th speaker, and I is the total number of speakers; the emotion label set E = {e1, e2,..., e j ,..., e M}, where M represents the number of emotion labels, and e jrepresent a certain class of emotional tags. Based on the speaker identity tags and the emotional class tags, all samples in the training data set are jointly grouped according to the binary combination of (speaker, emotion), and each group constitutes a meta-class, that is, the meta-class c ij = (s i , e j ), where s i ∈ S, e j ∈ E, that is, the jth emotion of the ith person is taken as the meta-class c ij In this embodiment, the total number of speakers I = 8, the emotional class E = {angry, happy, neutral, sad}, that is, M = 4.

[0058] Step 2.2) Repeat the grouping method in step 2.1) to obtain the set of all meta-classes and the corresponding meta-class tags, and the total number of meta-classes is I x M. In this embodiment, I = 8, M = 4, and the total number of meta-classes is 4 x 8 = 32; wherein the meta-class tag {′Ses01M′, ′angry′} represents the speech sample of the speaker ′Ses01M′ with the emotion ′angry′.

[0059] Step 3) Meta-model construction: input the 768-dimensional speech emotion feature vector X into the fully connected neural network, and the number of network layers is 2; the first layer is a pre-feature compression layer, which is composed of a fully connected layer with an input dimension of 768 and a ReLU activation function, and realizes compression of the input feature from 768 dimensions to F dimensions, F takes a value in the range of 200-350, and outputs an F-dimensional feature h; the second layer is a classification output layer, which is composed of a fully connected layer and a softmax activation function, and maps the F-dimensional feature to an M-dimensional meta-class space, and outputs the probability distribution P(y|X) = Softmax(W out h+b out )

[0060] wherein P(y|X) represents the probability of the output label y given the input speech emotion feature vector X, W out is an M x F weight matrix, and b out is a bias. The final recognition result y * is the label y * with the maximum probability, that is, y

[0061] In this embodiment, F = 256, M = 4; the first layer fully connected layer has an input of 768 nodes and an output of 256 nodes; h is a 256-dimensional feature; the second layer fully connected layer has an input of 256 nodes and an output of 4 nodes, each node representing a class, and can realize 4 classification.

[0062] Step 4) Metamodel parameter initialization: Initialize the metamodel constructed in step 3) using f θ Let θ represent the meta-model parameters, and let θ be randomly initialized.

[0063] In this embodiment, the meta-model parameters include the parameters of two fully connected layers.

[0064] Step 5) Outer Loop Update: The meta-model parameters are updated through the outer loop, with each outer loop refining the parameters. This includes five steps: generating the meta-task, inner loop update, calculating the learning rate, updating the meta-model parameters, and determining whether to stop. Each step is explained in detail below:

[0065] Step 5.1) Generate meta-tasks: Using a random sampling strategy, generate meta-tasks from the meta-category set C obtained in Step 2); set the meta-task complexity to M, that is, each meta-task contains M meta-categories, and the meta-task complexity should be consistent with the final number of sentiment categories to be identified; randomly select M meta-categories from C to form one meta-task T. i For each selected meta-category, all samples of that meta-category are extracted from the speech emotion feature vector set as the training set for that meta-task. Simultaneously, the meta-task T... i All samples were randomly shuffled; as shown in the attached document. Figure 2 As shown, a total of K meta-tasks are generated, and each meta-task has M categories, denoted by T1, T2, ..., T... K This approach uses the binary combination of (speaker, emotion) as a meta-category and forms a meta-task, enabling the model to learn both speaker and emotion information simultaneously when processing the meta-task. It focuses on the intrinsic connections and differences between speaker identity and emotion information, strips away speaker features, focuses on emotion features, and improves the accuracy of cross-speaker emotion recognition.

[0066] In this embodiment, K=5, and a total of 5 meta-tasks T1, T2, ..., T5 are generated, with each meta-task having 4 categories.

[0067] Step 5.2) Inner loop update: as shown in the appendix Figure 3 As shown, the process includes five steps: extracting the meta-task, copying the meta-parameters, updating the parameters, calculating the parameter difference, and determining whether to terminate. A detailed explanation of each step is as follows:

[0068] Step 5.2.1) Extract meta-tasks: Extract the meta-tasks T1, T2, ..., T generated in Step 5.1) sequentially according to the number of inner loop iterations. i ,...T K The i-th inner loop extracts the meta-task T i As a training set.

[0069] Step 5.2.2) Copy meta-parameters: Copy the current meta-model parameter θ to the current task-specific meta-model: φ i ←θ, φ i For the current meta-task T i The corresponding meta-model parameters yield the meta-model specific to the current meta-task.

[0070] Step 5.2.3) Parameter Update: Update the parameters for the current meta-task T. i The cross-entropy loss function is calculated using the following formula:

[0071]

[0072] Where J is the current metatask T i The number of samples, X j It is the current task T i The speech emotion feature vector of the j-th sample in the dataset. It is X j Input into the current task-specific metamodel The output of the m-th node in the subsequent output layer. j，m For the corresponding classification label, if the j-th sample belongs to the m-th class, then y j，m =1, otherwise y j，m =0.

[0073] Perform backpropagation to calculate gradient Update the parameter φ of the current meta-task-specific meta-model. i for

[0074] Where β is the learning rate of the inner loop. In this embodiment, β = 0.01.

[0075] Step 5.2.4) Calculate the parameter difference: Calculate the parameter φ of the updated current meta-task-specific meta-model. i The difference Δθ between the current metamodel parameter θ and the current metamodel parameter θ i :

[0076] Δθ i =φ i -θ

[0077] Step 5.2.5) Determine if the loop has ended: Check if the loop has run K times. If yes, exit the inner loop and execute step 5.3). If no, repeat steps 5.2.1) to 5.2.5) until the loop has run K times. Extract K meta-tasks T1, T2, ..., T through the inner loop. K And perform an inner loop update for each meta-task, calculating the loss. and the corresponding parameter differences Δθ1, Δθ2, ..., Δθ K .

[0078] In this embodiment, K = 5, each outer loop repeats the inner loop 5 times, the first inner loop gets and Δθ1, the second inner loop gets and Δθ2, the third inner loop gets and Δθ3, the fourth inner loop gets and Δθ4, the fifth inner loop gets and Δθ5.

[0079] Step 5.3) Calculate learning rate: update the learning rate a of the outer loop with the cosine annealing learning strategy, the formula is

[0080]

[0081] where a0 is the initial value, t is the number of outer loops, if it is the nth outer loop, then t = n, N is the total annealing period, that is, the total number of outer loops. a will change dynamically with the increase of the number of outer loops t.

[0082] In this example, the initial value a0 = 0.1, and the total number of outer loops N = 150.

[0083] Step 5.4) Update metamodel parameters: update the metamodel parameters K and the learning rate a obtained in step 5.3) according to the parameter differences Δθ1, Δθ2,..., Δθ

[0084]

[0085] Step 5.5) Determine whether to stop: calculate the loss of the outer loop according to the loss obtained in step 5.2)

[0086]

[0087] Determine whether the average loss change |ΔL| of the continuous W outer loops is less than the threshold value δ, if yes, stop the outer loop, and take θ (t) as the optimal metamodel parameter θ * ; if not, determine whether the outer loop has been repeated N times, if yes, stop the outer loop, and take θ (t) as the optimal metamodel parameter θ * , if not, let θ (t) be the metamodel parameter θ of the next outer loop, that is, θ←θ (t) , continue to repeat steps 5.1)-5.5). As shown in the attached Figure 4 , the metamodel parameters are updated once every time the outer loop is passed.

[0088] In this embodiment, W = 10, and the threshold value δ = 0.001. After the 96th outer loop, that is, after repeating steps 5.1) - 5.5) 96 times, the average loss change | ΔL | of the last 10 outer loops is less than 0.001, that is, the average loss change of the 87th - 96th outer loop is less than 0.001, so the outer loop is stopped, and θ (96) As the optimal meta-model parameter θ * Step 6) is performed again.

[0089] Step 6) target task adaptation: the target task is to perform speech emotion recognition, which can recognize E = {e1, e2,..., e j ,..., e M} M types of emotions. It includes three steps of training data preparation, target task specific meta-model fine-tuning, and target task recognition.

[0090] Step 6.1) training data preparation: use speech data with E = {e1, e2,..., e j ,..., e M} M emotion labels as target task training data. Extract 768-dimensional speech emotion feature vectors for each speech according to step 1). Since meta-learning has the ability to learn to learn, the number of classes of the target task needs to be consistent with the number of meta-classes in step 5.1), but the specific emotion categories in the target task can be different from the emotions in step 2).

[0091] In this embodiment, the target task is to recognize E = {angry, happy, neutral, surprise} 4 types of emotions. The number of classes of the target task is consistent with the number of meta-classes of the meta-task, which is 4. The specific emotion categories in the target task are angry, happy, neutral, and surprise, which are different from the emotion categories in step 2) angry, happy, neutral, and sad.

[0092] Step 6.2) target task specific meta-model fine-tuning: load the optimal meta-model parameter θ * , and input the target task training data to the optimal meta-model to calculate the cross-entropy loss as

[0093]

[0094] where T is the number of target task training data, X t is the speech emotion feature vector of the tth sample in the target task training data, is the input of X t to the optimal meta-model Output of the mth node of the output layer. t,m For the corresponding classification label, if the tth sample belongs to the mth class, y t，m = 1, otherwise y t，m = 0.

[0095] In this embodiment, the training data amount of the target task is 100, and the corresponding speech of each emotion category is 25. The meta-model The output layer has 4 nodes. The 1st node outputs the probability of X t belonging to angry, and this probability value is represented by p (X | angry) ; the 2nd node outputs the probability of X t belonging to happy, and this probability value is represented by p (X | happy) ; the 3rd node outputs the probability of X t belonging to neutral, and this probability value is represented by p (X | neutral) ; and the 4th node outputs the probability of X t belonging to surprise. Through this probability, this probability value is represented by p (X | surprise).

[0096] The gradient is calculated by performing back propagation, and the parameters θ # of the optimal meta-model are fine-tuned to obtain the parameters θ

[0097]

[0098] Step 6.3) Target task recognition: according to step 1), the 768-dimensional speech emotion feature vector of the speech to be recognized is extracted, and the speech emotion feature vector is input into the target task exclusive meta-model to obtain the probability corresponding to M emotion categories, and the emotion with the maximum probability is taken as the emotion recognition result, and the speech emotion recognition is completed.

[0099] In this embodiment, after the 768-dimensional speech emotion feature vector of the speech to be recognized is input into the target task exclusive meta-model , the output probability corresponding to 4 emotion categories is [0.1, 0.2, 0.1, 0.6], and the maximum 0.6 is the probability of the speech belonging to surprise, that is, the emotion recognition result is surprise.

[0100] The method described in the present application can be understood by those skilled in the art that the above-mentioned speech emotion recognition method can be completed by a program, and the program can be stored in a computer readable storage medium.

[0101] The above merely provides one specific embodiment of the present application, and obviously cannot be used to limit the scope of the present application, and equivalent changes made according to the claims of the present application shall still fall within the scope of the present application.

Claims

1. A method for cross-speaker small sample speech emotion recognition based on meta-learning, characterized in that, The six steps include voice feature extraction, meta-class generation, meta-model construction, meta-model parameter initialization, outer loop update, and target task adaptation. Step 1) Voice feature extraction: emotion2vec model is used for voice emotion feature extraction, and emotion2vec model is used to extract 768-dimensional voice emotion feature vector X from each input voice; IEMCOAP dataset is used for experiment, which is a widely used dataset for voice emotion recognition; Step 2) Meta-class generation: a dual grouping mechanism based on emotion features and speaker identity is adopted, and by combining emotion features and speaker information, a (speaker, emotion) binary combined class is constructed to realize fine-grained meta-learning task division; Step 3) Meta-model construction: input the 768-dimensional voice emotion feature vector X in step 1) into the fully connected neural network, and the number of layers of the fully connected neural network is 2; The first layer is a pre-feature compression layer composed of an input dimension of 768-dimensional fully connected layer and ReLU activation function, which realizes the compression of input features from 768-dimensional to F-dimensional, F takes the value range of 200-350, and outputs F-dimensional features h; The second layer is a classification output layer composed of a fully connected layer and a softmax activation function, which maps F-dimensional features to M-dimensional meta-class space and outputs the probability distribution of each meta-class: P(y | X) = Softmax(W out h + b out ) where P(y|X) represents the probability of the output label y given the input speech emotion feature vector X, W out is a MxF-dimensional weight matrix, b out is a bias; the final recognition result y * is the label with the largest probability: y * = argmax P(y | X) Step 4) Meta-model parameter initialization: The meta-model constructed in step 3) is denoted as f θ where θ represents the meta-model parameters, which are randomly initialized. Step 5) Meta-model parameter updating: The meta-model parameters θ are updated by minimizing the loss function L(θ) using the gradient descent method. Step 5) Outer loop update: update the meta-model parameters through outer loop, update the parameters once in each outer loop; Including generating meta-task, inner loop update, calculating learning rate, meta-model parameter update, and judging whether to stop five steps; Step 6) Target task adaptation: The target task is to perform speech emotion recognition, identifying E = {e1, e2,..., eM} M classes of emotions. j ,..., e M}M classes of emotions.

2. The method of claim 1, wherein the method is based on meta-learning. Step 2) includes the following steps: Step 2.1) Define the speaker set S = {s1, s2, ..., s...} i , ..., s I }, where s i Let I represent the i-th speaker, and I be the total number of speakers; the sentiment tag set E = {e1, e2, ..., e...} j , ..., e M }, where M represents the number of sentiment tag categories, e j Representing a certain type of sentiment label; based on the speaker identity label and sentiment category label, all samples in the training dataset are jointly grouped according to the binary combination of (speaker, sentiment), and each group constitutes a meta-category, i.e., defining meta-category c. ij =(s i e j ), where s i ∈S, e j ∈E, which means taking the j-th emotion of the i-th person as the meta-category c. ij ; Step 2.2) Repeat the grouping method in step 2.1) to get all meta-category sets as and the corresponding meta-category labels, with a total of I x M meta-categories.

3. The method of claim 1, wherein the method is based on meta-learning. Step 5) specifically includes: Step 5.1) generating meta-tasks: using random sampling strategy, generating meta-tasks from the meta-category set C obtained in step 2); setting the meta-task complexity M, i.e. the number of meta-categories contained in each meta-task is M, and the meta-task complexity is consistent with the number of sentiment categories to be finally identified; randomly selecting M meta-categories from C to form a meta-task T i For each selected meta-category, extracting all samples of the meta-category from the speech emotion feature vector set as the training set of the meta-task, and randomly shuffling all samples in the meta-task T i , a total of K meta-tasks are generated, each with M categories, denoted as T1, T2,..., T K ; Step 5.2) Inner loop update: including extracting meta-task, copying meta-parameter, parameter update, calculating parameter difference, and judging whether to end five steps; The detailed description of each step is as follows: Step 5.2.1) Extracting meta-tasks: Extract the meta-tasks T1, T2,..., Ti in step 5.1) in turn according to the number of inner loops, and take the meta-task Ti extracted in the i-th inner loop as the training set. i ,... Ti K , the meta-task Ti extracted in the i-th inner loop as the training set. i ​ Step 5.2.2) Copy meta-parameters: Copy the current meta-model parameters θ to the current task-specific meta-model: φ i ← θ, φ i for the current meta-task T i corresponding meta-model parameters, resulting in the current task-specific meta-model Step 5.2.3) Parameter update: Update the current meta-task T i The cross-entropy loss function is calculated, which is where J is the current meta-task T i , the number of samples, X j is the speech emotion feature vector of the jth sample in the current task T i , y is the output of the mth node in the output layer after inputting X j to the current task-specific meta-model , and y j，m is the corresponding classification label, where y j,m = 1 if the jth sample belongs to the mth class, otherwise y j,m = 0. Performing backpropagation to compute gradients updating the parameters φ of the current meta-task-specific meta-model i for Where β is the learning rate of inner loop; In this embodiment, β = 0.01; Step 5.2.4) Calculate parameter difference: Calculate the parameter difference of the updated current meta-task specific meta-model φ i Difference Δθ from current meta-model parameter θ i : Δθ i = φ i - θ Step 5.2.5) judging whether to end: judging whether K times have been looped, if yes, breaking the inner loop and executing step 5.3), if not, repeating steps 5.2.1) to 5.2.5) until K times are looped; extracting K meta-tasks T1, T2,..., TK through the inner loop K , and performing inner loop updating on each meta-task to calculate the loss and the corresponding parameter difference Δθ1, Δθ2,..., Δθ K ; Step 5.3) Calculate learning rate: use cosine annealing learning strategy to update the learning rate α of outer loop, the formula is Where α0 is the initial value, t is the number of outer loop, if it is the n-th outer loop, then t = n, N is the total annealing period, that is, the total number of outer loop; α will change dynamically with the increase of the number of outer loop t; Step 5.4) Meta-model parameter update: Update the meta-model parameters according to the parameter differences Δθ1, Δθ2,..., Δθ K and the learning rate α obtained in steps 5.2) and 5.3) Step 5.5) Stopping criterion: loss obtained in step 5.2) Computing outer loop average loss variation whether the average loss variation |ΔL| of the continuous W outer loops is less than the threshold value δ, if yes, stop the outer loop, and set θ (t) as the optimal meta-model parameter θ * ; if no, whether the outer loop has been repeated for N times, if yes, stop the outer loop, and set θ (t) as the optimal meta-model parameter θ * , if no, set θ (t) as the meta-model parameter θ of the next outer loop, i.e. θ←θ (t) , and continue to repeat steps 5.1)-5.5), and the meta-model parameter is updated every time the outer loop is passed.

4. The method of claim 3, wherein the method is based on meta-learning. In step 5.3) Calculate learning rate, the initial value α0 = 0.1, and the total number of outer loop N = 150.

5. The method of claim 3, wherein the method is based on meta-learning. Step 6) includes three steps of training data preparation, target task specific meta-model fine-tuning, and target task recognition; Step 6.1) Training data preparation: Speech data with E = {e1, e2,..., e j ,...,e M}M class emotion labels are used as target task training data; 768-dimensional speech emotion feature vectors of each speech are extracted according to step 1); Step 6.2) Target task-specific meta-model fine-tuning: Load optimal meta-model parameters θ * The training data of the target task is fed into the optimal meta-model The cross-entropy loss is calculated as where T is the number of training data of the target task, X t is the speech emotion feature vector of the t-th sample in the target task training data, is the output of the m-th node of the optimal meta-model t input layer; y is the output of the m-th node of the optimal meta-model t，m output layer; y t，m = 1 if the t-th sample belongs to the m-th class, otherwise y t，m = 0. Performing backpropagation to compute gradients Fine-tuning the optimal meta-model to obtain the parameters θ of the target task-specific meta-model # For Step 6.3) target task recognition: according to step 1), the 768-dimensional speech emotion feature vector is extracted from the speech to be recognized, and the speech emotion feature vector is input into the target task-specific meta-model to obtain the probability corresponding to M types of emotions, and the emotion with the maximum probability is the result of emotion recognition, and the speech emotion recognition is completed.

Images