Multi-modal emotion recognition method based on label signal guidance

Abstract

The invention discloses a multi-modal emotion recognition method based on label signal guidance. A multi-modal emotion recognition model comprising a feature extraction module, a cross-modal interaction module, a label enhancement module and a dynamic fusion and classification module is constructed. By introducing an emotion label signal, effective interaction with audio and text features is carried out, and fusion and extraction of emotion features are optimized. Expression of modal features is guided through a label enhancement module, and the updating process of label signals is smoothed in combination with a moving average mechanism, so that the model is more stable and accurate in the emotion category classification process. Meanwhile, the invention further provides a joint target optimization method, and the consistency between the emotion label and the multi-modal features is enhanced by combining attribution prediction consistency constraint (APC) and a cross entropy loss function, so that the emotion recognition precision is further improved.

Description

Technical Field

[0001] The present invention relates to the field of speech emotion recognition, and relates to, but is not limited to, a multimodal emotion recognition method guided by label signals. Background Art

[0002] Emotion recognition technology aims to analyze emotional states by integrating various modal information, which is an important research direction in the field of human-computer interaction and plays an important role in enhancing user experience, optimizing interaction processes, and assisting emotional analysis.

[0003] Emotions are multimodal, and it is difficult for a single modality to comprehensively capture the diversity and complexity of emotions. Reasonably modeling multimodal information is an effective way to improve recognition performance. Due to its advantages in information capture and cross-modal correlation modeling, the attention mechanism has become one of the mainstream methods widely used in multimodal emotion recognition. Many studies have proposed innovations based on the attention mechanism and achieved remarkable progress. Although existing methods have made certain progress in cross-modal interaction and emotion recognition, most studies have ignored the role of emotion labels, resulting in the model failing to fully utilize emotion information during feature extraction and information integration, thus limiting the improvement of overall performance. Emotion labels are not only the output results of classification tasks, but also contain deep information about emotional states. In recent years, some studies have begun to explore the application of emotion label information in emotion recognition. For example, using a pre-trained text model to extract label text features, and optimizing the distribution of audio features and label text features through contrastive learning to make the audio features with the same emotion label closer, while further separating the audio features with different emotion labels. However, directly using label text as guidance information provides highly simplified emotional descriptions, which cannot capture more complex emotional details or subtle emotional changes. Therefore, how to make full use of emotion labels is one of the important issues in this field. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a multimodal emotion recognition method guided by label signals, aiming to introduce emotion label signals as additional input information for emotion recognition, and capable of providing key information required for emotion classification through label signals.

[0005] The technical solution of the embodiments of the present invention is specifically as follows:

[0006] Embodiments of the present invention provide a multimodal emotion recognition method guided by label signals, the method comprising:

[0007] Obtain a multi-modal sentiment data sample set, where each sample contains associated audio information and text information; construct a multi-modal sentiment recognition model, including a feature extraction module, a cross-modal interaction module, a label enhancement module, a dynamic fusion and classification module; wherein, the feature extraction module includes a first branch for extracting initial text features and a second branch for extracting initial audio features; the cross-modal interaction module is used to perform cross-modal attention calculation on the initial text features and the initial audio features to obtain first text features and first audio features; the label enhancement module is used to initialize the label information as a label embedding and use it as a query vector to enhance the first text features and the first audio features respectively, and perform weighted fusion of the enhanced features with their respective corresponding initial features to obtain final text features and final audio features; the dynamic fusion and classification module is used to perform dynamic fusion on the final audio features and the final text features using an expert gating fusion mechanism and output the prediction probabilities of each emotion category; based on the sentiment data samples, iteratively train the multi-modal sentiment recognition model using the designed joint objective optimization loss; wherein, the joint objective optimization loss is obtained by combining the attribution prediction consistency loss and the cross-entropy loss function; identify the sentiment information to be recognized through the trained multi-modal sentiment recognition model to obtain a sentiment prediction result.

[0008] In some embodiments, in the first branch, the text features of the input text information are first extracted by Roberta, and the text features further capture the context information in the sequence through a Bi-LSTM layer to obtain the initial text features H t0 ; in the second branch, WavLM is used to extract the first features of the input audio information, and at the same time, the mel spectrogram of the audio information is extracted. The reshaped spectrogram image is extracted by a pre-trained AlexNet to obtain the second features, and the first features and the second features are added to obtain the initial audio features H a0 .

[0009] In some embodiments, the cross-modal interaction module uses one of the initial text features and the initial audio features as a query vector respectively, and uses the other as a key and a value for attention calculation, and correspondingly obtains the first text features H t1 containing audio information and the first audio features H a1 containing text information.

[0010] In some embodiments, in the label enhancement module, the label embedding is used as the query vector Q, and the first audio features H a1 are used as the key K a-t and the value V a-t , and attention calculation is performed to obtain the second audio features H a2 ; at the same time, the first text features H t1 are used as the key Kt-a Sum value V t-a , perform attention calculation to obtain the second text feature H t2 ; and obtain the final audio feature H a and the final text feature H t :

[0011] H a = sigmoid(H a2 ) ⊙ H a0 + H a2 ;

[0012] H t = sigmoid(H t2 ) ⊙ H t0 + H t2 ;

[0013] In the formula, ⊙ represents element-wise multiplication; H a0 , H t0 are the initial audio feature and the initial text feature respectively; sigmoid() is the activation function.

[0014] In some embodiments, during the model training process, the label enhancement module updates the label embedding in the following manner and uses a moving average mechanism to control the update rate:

[0015] L t = αL t-1 + (1 - α)L' t-1 ;

[0016] In the formula, α is a hyperparameter for controlling the update rate, L t-1 is the label embedding at the start of the (t - 1)-th training epoch, L' t-1 is the label embedding after updating in the (t - 1)-th training epoch, and L t is the label embedding at the start of the t-th training epoch; when t = 1, the label embedding is randomly initialized.

[0017] In some embodiments, the dynamic fusion of the final audio feature and the final text feature includes: concatenating the final audio feature and the final text feature into a combined feature vector, and calculating the respective dynamic weights of the final audio feature and the final text feature by passing the combined feature vector through a fully connected layer: weighting the final audio feature and the final text feature based on the dynamic weights to obtain the final fused feature representation; feeding the fused feature representation into a multi-layer perceptron to obtain the emotion category score of the sample; using the softmax function to normalize the emotion category score of each sample to obtain the probability that the sample belongs to each emotion category.

[0018] In some embodiments, the joint target optimization loss is represented by the following formula:

[0019]

[0020] L(θ) = αL ce (θ) + βL apc (θ);

[0021] where L apc (θ) is the APC loss, L ce (θ) is the cross - entropy loss, L(θ) is the joint target optimization loss, α and β are the balance weights of the cross - entropy loss and the APC loss respectively, M is the total number of samples in the entire batch, N is the total number of sentiment categories, p i,j is the predicted probability that the i - th sample belongs to the j - th emotion category, q i,j is the similarity probability between the i - th sample and the j - th emotion category, is the JS divergence between the similarity distribution and the predicted probability distribution, y i,j is the true label.

[0022] The beneficial effects brought by the technical solutions provided in the embodiments of the present invention at least include:

[0023] In the embodiments of the present invention, a multi - modal sentiment recognition model including a feature extraction module, a cross - modal interaction module, a label enhancement module, a dynamic fusion and classification module is constructed. By introducing sentiment label information, this model effectively interacts with audio and text features, optimizing the fusion and extraction of sentiment features. Among them, the label enhancement module guides the expression of modal features and, at the same time, combines the moving average mechanism to smooth the update process of the label signal, making the model more stable and accurate in the classification process of sentiment categories. Meanwhile, the present invention also proposes a joint target optimization method. By combining the attribution prediction consistency constraint and the cross - entropy loss function, the consistency between sentiment labels and multi - modal features is strengthened, thereby further improving the accuracy of sentiment recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to these drawings, where:

[0025] Figure 1 is a schematic flowchart of the multi - modal sentiment recognition method guided by label signals provided in the embodiments of the present invention;

[0026] Figure 2It is the overall structure diagram of the multi-modal emotion recognition model provided by the embodiments of the present invention;

[0027] Figure 3 It is a schematic diagram of the calculation process of the attribution prediction consistency loss provided by the embodiments of the present invention. Specific embodiments

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0029] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0030] It should be noted that the terms "first / second / third" involved in the embodiments of the present invention are only used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first / second / third" can be interchanged with a specific order or sequence when permitted, so that the embodiments of the present invention described here can be implemented in an order other than that illustrated or described here.

[0031] Those skilled in the art of the present technology can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used here have the same meaning as the general understanding of those of ordinary skill in the technical field to which the embodiments of the present invention belong. It should also be understood that terms such as those defined in a general dictionary should be understood to have a meaning consistent with the meaning in the context of the prior art, and will not be interpreted with an idealized or overly formal meaning unless specifically defined as here.

[0032] Figure 1 It is a schematic flow diagram of a multi-modal emotion recognition method guided by a label signal provided by the embodiments of the present invention. As Figure 1 shown, the method at least includes the following steps:

[0033] Step S110, obtain a multi-modal emotion data sample set, where each sample includes associated audio information and text information.

[0034] Here, several groups of associated audio information and text information, as well as the corresponding sentiment labels for each group of information, are collected. The text information can be conversations, comments, etc., and the audio information can be voices, background music, etc. For example, a sample includes the text information "Hello" and the corresponding audio information of a segment of "Hello".

[0035] Step S120, construct a multi-modal sentiment recognition model, including a feature extraction module, a cross-modal interaction module, a label enhancement module, a dynamic fusion and classification module.

[0036] Here, the feature extraction module includes a first branch for extracting initial text features and a second branch for extracting initial audio features; the cross-modal interaction module is used to perform cross-modal attention calculation on the initial text features and the initial audio features to obtain the first text features and the first audio features; the label enhancement module is used to initialize the label information into label embeddings and use them as query vectors to enhance the first text features and the first audio features respectively, and perform weighted fusion of the enhanced features with their respective corresponding initial features to obtain the final text features and the final audio features; the dynamic fusion and classification module is used to perform dynamic fusion on the final audio features and the final text features using an expert gating fusion mechanism and then output the prediction probabilities of each emotion category. The overall structure diagram of the multi-modal sentiment recognition model is as Figure 2 shown.

[0037] In the feature extraction module, the first branch can use a pre-trained text embedding model (such as BERT, GPT, etc.) or traditional methods such as TF-IDF and bag-of-words models to convert the text into a high-dimensional vector representation. The second branch can use an audio processing library (such as Librosa) to extract audio features, such as Mel-frequency cepstral coefficients (MFCC), fundamental frequency (pitch), energy, etc.

[0038] The cross-modal interaction module uses an attention mechanism (such as multi-head self-attention, cross-attention, etc.) to calculate the correlation between the text and audio features. The features are weighted according to the attention weights to obtain the first text features and the first audio features that fuse cross-modal information.

[0039] The label enhancement module initializes label information (such as emotion categories like "happy", "angry", "sad", etc.) into a set of label embedding vectors. Label embedding not only provides additional emotion category information for emotion classification but also establishes associations with audio features and text features respectively, enabling the model to focus on key features related to emotions. This module optimizes the fusion and extraction of emotion features through introducing emotion label information and effectively interacting with the first audio feature and the first text feature. Finally, the final text feature and the final audio feature have fused label information. It should be noted that the label embedding vectors are learnable. The first initialization is a set of random numbers generated. During the model training process, the label embedding will be adjusted and updated along with the current batch of data and used as the input for the next batch. To avoid the training process from becoming unstable, a moving average method can be used to smooth the update process of the label embedding.

[0040] In the dynamic fusion and classification module, the expert gating mechanism can dynamically adjust its weights according to the importance of each modality. The fused features are input into a classifier (such as a fully connected layer, a softmax layer, etc.) to output the predicted probabilities of each emotion category.

[0041] Step S130, based on the emotion data samples, iteratively train the multi-modal emotion recognition model using the designed joint objective optimization loss.

[0042] Here, the joint objective optimization loss is obtained by combining the attribution prediction consistency loss APC and the cross-entropy loss function. Feature fusion can effectively make up for the deficiencies of a single modality. However, regarding how to ensure that the fused features can effectively represent emotion category information, there is a lack of effective supervision signals to ensure that this fusion can bring better emotion classification effects. To solve this problem, the embodiments of the present invention propose a joint objective optimization that combines the attribution prediction consistency (APC) loss and the cross-entropy loss to jointly guide the model training.

[0043] The attribution prediction consistency constraint (APC) is a method for strengthening the prediction consistency of the model. In the emotion recognition task, APC helps to ensure the consistency between the feature extraction and label prediction of the model in different modalities (such as text, audio, video, etc.). This consistency is crucial for improving the accuracy of emotion recognition because it can reduce the prediction errors caused by differences between modalities. The APC loss not only enhances the supervision role of label information during the training process by minimizing the difference between the similarity distribution between the fused feature representation and the label embedding and the model prediction distribution but also enables the fused features to be more precisely aligned with emotion categories, thereby improving the accuracy, fineness, and robustness of emotion classification.

[0044] The cross-entropy loss function is one of the commonly used loss functions in machine learning, which is used to measure the difference between the probability distribution predicted by the model and the true label distribution. In the sentiment recognition task, the cross-entropy loss function can reflect the accuracy of the model in predicting sentiment labels. When the predicted distribution is closer to the true distribution, the cross-entropy loss is smaller, indicating that the prediction result of the model is more accurate.

[0045] The joint objective optimization method provided by the embodiments of the present invention strengthens the consistency between the sentiment labels and the multi-modal features, thereby further improving the accuracy of sentiment recognition.

[0046] Step S140, identify the sentiment information to be identified through the trained multi-modal sentiment recognition model to obtain a sentiment prediction result.

[0047] Here, load the already trained multi-modal sentiment recognition model from the storage location. This model should be able to process data of multiple modalities and output a sentiment prediction result. Input the sentiment information to be identified into the model. This usually includes using the pre-processed data (such as text features, audio features, etc.) as the input of the model. After receiving the input data, the model will perform a series of calculations and reasoning processes. This includes the transfer of data between different layers, the fusion of features, and the final sentiment prediction.

[0048] Furthermore, a set of test data with known sentiment labels can also be used to evaluate the performance of the model. This can be done by calculating metrics such as accuracy, recall rate, F1 score, etc. According to the evaluation results, the model can be further adjusted and optimized to improve its performance.

[0049] In the embodiments of the present invention, a multi-modal sentiment recognition model including a feature extraction module, a cross-modal interaction module, a label enhancement module, a dynamic fusion and classification module is constructed. By introducing a sentiment label signal, this model effectively interacts with audio and text features, optimizing the fusion and extraction of sentiment features. Among them, the label enhancement module guides the expression of modal features, and at the same time combines a moving average mechanism to smooth the update process of the label signal, making the model more stable and accurate in the classification process of sentiment categories. At the same time, the present invention also proposes a joint objective optimization method, which strengthens the consistency between the sentiment labels and the multi-modal features by combining the attribution prediction consistency constraint and the cross-entropy loss function, thereby further improving the accuracy of sentiment recognition.

[0050] In some embodiments, in the first branch, first extract the text features of the input text information through Roberta, and the text features further capture the context information in the sequence through a Bi-LSTM layer to obtain the initial text feature H t0; In the second branch, the first feature of the input audio information is extracted using WavLM, and at the same time, the Mel spectrogram of the audio information is extracted. The reshaped spectrogram image is used to extract the second feature through the pre-trained AlexNet. The first feature and the second feature are added together to obtain the initial audio feature H a0 。

[0051] Here, Roberta (Robustly optimized BERT pretraining approach) is a pre-trained deep learning model based on the Transformer architecture, which is particularly suitable for natural language processing tasks. After the input text information, the Roberta model encodes it and extracts deep text features. These features capture the semantic and syntactic information of the text. The bidirectional long short-term memory network (Bi-LSTM) can capture the context information in the text sequence. The text features extracted by Roberta are input into the Bi-LSTM layer, and through the combination of forward and backward LSTMs, the context dependencies in the text sequence are further extracted to generate more refined initial text features.

[0052] WavLM is a pre-trained model specifically designed for audio processing. After the input audio data, WavLM can extract the high-level features of the audio, which capture the semantic and temporal information in the audio signal. The Mel spectrogram is a method of converting the audio signal into a frequency-domain representation, which is particularly suitable for capturing the subtle changes in the audio signal. The extracted Mel spectrogram is converted into a spectrogram image for further processing.

[0053] AlexNet is a classic convolutional neural network architecture commonly used in image recognition tasks. After reshaping the spectrogram image to a size suitable for the input of AlexNet, AlexNet can extract the image features in the spectrogram, which reflect the visual representation of the audio signal. The audio features extracted from WavLM and AlexNet are added together or fused in other forms to obtain richer and more comprehensive initial audio features. This fusion method combines the high-level semantic features of the audio signal and the visual features of the spectrogram, improving the expressive power and robustness of the audio features.

[0054] This feature extraction module realizes the comprehensive feature extraction of text and audio data by combining deep learning models such as Roberta, Bi-LSTM, WavLM, and AlexNet. The text features capture semantic and context information through the combination of Roberta and Bi-LSTM, while the audio features capture high-level semantic and visual features through the combination of WavLM and AlexNet. This multi-modal feature extraction method helps to improve the accuracy and robustness of emotion recognition.

[0055] In some embodiments, the cross-modal interaction module uses one of the initial text feature and the initial audio feature as the query vector respectively, and uses the other as the key and value for attention calculation, and correspondingly obtains the first text feature H containing audio information t1 and the first audio feature H containing text information a1 .

[0056] Here, as Figure 2 shown, on the one hand, the cross-modal interaction module takes the initial text feature H t0 as the query Q t , the initial audio feature H a0 as the key and value, and calculates the attention weight of each key according to the similarity (such as dot product, cosine similarity, etc.) between the query vector and the key vector. Then, these weights are used to perform weighted summation on the value vector to obtain the first text feature H containing audio information t1 ; on the other hand, the initial audio feature H a0 is taken as the query Q a , the initial text feature H t0 is taken as the key and value, and the attention calculation is also performed to obtain the first audio feature H containing text information a1 .

[0057] When the cross-modal interaction module processes multi-modal data (such as audio and text), it effectively fuses the information of different modalities through the attention mechanism, thereby improving the model's ability to understand and process data. Through bidirectional cross-modal attention calculation, the model can understand the relationship between them from both the text and audio perspectives, so as to more comprehensively capture the information in the data. In the emotion recognition task, audio and text often contain complementary information, and the cross-modal interaction module can use this information to improve the accuracy of emotion recognition.

[0058] In some embodiments, in the label enhancement module, the label embedding is used as the query vector Q, and the first audio feature H a1 is used as the key K a-t and the value V a-t , and the attention calculation is performed to obtain the second audio feature H a2 ; at the same time, the first text feature H t1 is used as the key K t-a and the value V t-a , and the attention calculation is performed to obtain the second text feature H t2 ; and the final audio feature H a and the final text feature H t are obtained through the following formula

[0059] h a = sigmoid(h a2 )⊙Ha0 +H a2 ;

[0060] H t = sigmoid(H t2 ) ⊙ H t0 +H t2 ;

[0061] wherein, ⊙ represents element-wise multiplication; H a0 and H t0 are the initial audio feature and the initial text feature respectively; sigmoid() is the activation function.

[0062] Here, in this way, the label embedding not only provides additional sentiment category information for the audio feature and the text feature, but also establishes associations with the audio feature and the text feature respectively, enabling the model to focus on the key features related to sentiment. For the obtained H a2 and H t2 and the initial audio feature H a0 and the text feature H t0 are weighted and fused, which can avoid the loss of original information.

[0063] In some embodiments, during the model training process, the label enhancement module updates the label embedding in the following manner and uses a moving average mechanism to control the update rate:

[0064] L t = αL t-1 + (1 - α)L' t-1 ;

[0065] wherein, α is a hyperparameter for controlling the update rate, L t-1 is the label embedding at the start of the (t - 1)-th training epoch, L' t-1 is the label embedding after update in the (t - 1)-th training epoch, and L t is the label embedding at the start of the t-th training epoch; when t = 1, the label embedding is randomly initialized.

[0066] Here, since the label embedding is learnable, the label embedding is adjusted according to the data of the current batch and used as the input for the next batch. Therefore, the label embeddings may vary significantly at the beginning of each training epoch, and the model needs to continuously adapt to the new label embeddings, making the training process unstable. To solve this problem, a moving average method is used to calculate the label embeddings. This update method can ensure that historical information can effectively guide the current learning process while avoiding excessive fluctuations in the label embeddings during training. During the update process of the label signal embedding vector, a moving average mechanism is used to control the update rate, greatly avoiding overfitting and information loss.

[0067] In some embodiments, the dynamic fusion of the final audio feature and the final text feature includes: concatenating the final audio feature and the final text feature into a combined feature vector, and calculating the respective dynamic weights of the final audio feature and the final text feature by passing the combined feature vector through a fully connected layer: weighting the final audio feature and the final text feature based on the dynamic weights to obtain the final fused feature representation; feeding the fused feature representation into a multi-layer perceptron to obtain the emotion category score of the sample; using the softmax function to normalize the emotion category score of each sample to obtain the probability that the sample belongs to each emotion category.

[0068] Here, the respective dynamic weights g a and g t of the final audio feature and the final text feature are calculated by the following formula:

[0069] g a , g t = softmax(w·H concat + b);

[0070] where softmax() is the normalization function, and H concat is the combined feature vector concatenated from the final audio feature H a and the final text feature H t .

[0071] Then, the final audio feature and the final text feature are weighted according to these weights to obtain the final fused feature representation H fused , where ⊙ represents element-wise multiplication: H fused = g a ⊙ H a + g t ⊙ H t ; The fused feature representation H fusedFeed it into a multi-layer perceptron to obtain the emotion category scores of each sample. Use the softmax function to normalize the scores of the samples to obtain the probability p that the sample belongs to each emotion category i,j , where i is the index of the sample and j is the index of the emotion category.

[0072] In some embodiments, the joint objective optimization loss is represented by the following formula:

[0073]

[0074] L(θ) = αL ce (θ) + βL apc (θ);

[0075] where, L apc (θ) is the APC loss, L ce (θ) is the cross-entropy loss, L(θ) is the joint objective optimization loss, α and β are the balance weights of the cross-entropy loss and the APC loss respectively, m is the total number of samples in the entire batch, n is the total number of emotion categories, p i,j is the predicted probability that the i-th sample belongs to the j-th emotion category, q i,j is the similarity probability between the i-th sample and the j-th emotion category, is the JS divergence between the similarity distribution and the predicted probability distribution, y i,j is the true label.

[0076] Here, the calculation process of the attribution prediction consistency loss is as Figure 3 shown. After the label information is vector-initialized, label embeddings are obtained. On the one hand, calculate the cosine similarity matrix between the label embeddings {L1,..., L j ,..., L n} and the fused features of the m samples in the entire batch, i.e., H fused = {H1,..., H i ,..., H m}. Then, apply the softmax function to normalize each cosine similarity score to obtain the similarity distribution q, where any element q i,j represents the similarity probability between the i-th sample and the j-th emotion label in the current batch, reflecting the possibility that the sample belongs to this emotion category.

[0077] On the other hand, for the entire batch of samples, calculate the predicted probability matrix through the emotion prediction probabilities output by the model, and apply the softmax function to normalize it to obtain the predicted probability distribution p, where any element p i,j is the predicted probability that the i-th sample belongs to the j-th emotion category.

[0078] Finally, the similarity distribution q and the predicted probability distribution p are compared, and the Jensen-Shannon divergence between the similarity distribution q and the predicted probability distribution p is minimized using the APC loss function to improve the consistency between the two. In this way, not only the supervisory role of the label information in the training process is enhanced, but also the fused features can be more accurately aligned with the sentiment categories.

[0079] Meanwhile, the cross-entropy loss L ce (θ) is calculated to estimate the quality of sentiment prediction. The APC loss and the cross-entropy loss are weighted and summed together to jointly form the joint objective optimization loss.

[0080] The above multi-modal sentiment recognition method guided by label signals will be described below in conjunction with a specific experimental example. However, it should be noted that this specific experimental example is only for better explaining the present invention and does not constitute an improper limitation of the present invention.

[0081] In this specific experimental example, the effectiveness of the method proposed by the present invention is tested on the IEMOCAP and MELD datasets. The IEMOCAP dataset contains approximately 12 hours of data, and all utterances are labeled with one of ten emotions: anger, happiness, sadness, neutral, frustration, excitement, fear, surprise, disgust, and other. The MELD dataset is a multi-party dataset created from "Friends". Each utterance is labeled with one of the following seven emotions: anger, disgust, sadness, joy, neutral, surprise, and fear.

[0082] The multi-modal sentiment recognition model guided by label signals proposed by the present invention is compared with some models that only rely on the attention mechanism on the two datasets. The comparison metrics include WA (weighted accuracy), UA (unweighted accuracy), and WF1 (weighted F1-score). The comparison results are shown in Tables 1 and 2 below.

[0083] Table 1 Comparison of model performances on the IEMOCAP dataset

[0084] Model Year Mode WA% UA% DWFORMER 2023 Audio 72.3 73.9 KS-Transformer 2022 Audio + Text 74.3 75.3 MMRBN 2024 Audio + Text 76.0 76.6 MFDR 2024 Audio + Text 75.7 77.0 The model proposed by the present invention 2024 Audio + Text 77.5 77.9

[0085] Table 2 Comparison of model performances on the MELD dataset

[0086] Model Year Mode WA% UA% DWFORMER 2023 Audio 72.3 73.9 KS-Transformer 2022 Audio + Text 74.3 75.3 MMRBN 2024 Audio + Text 76.0 76.6 MFDR 2024 Audio + Text 75.7 77.0 The model proposed by the present invention 2024 Audio + Text 77.5 77.9

[0087] As can be seen from Table 1 and Table 2, the results of the multi-modal emotion recognition model proposed by the present invention are better than those of other models on both datasets. It can be considered that the performance improvement is mainly attributed to the guiding role of label embedding in the model. Different from traditional models that mainly rely on the combination of audio and text features, the multi-modal emotion recognition model introduces label embedding into the model and further strengthens the guiding role of labels for audio and text modalities in various ways, significantly improving the performance of the model.

[0088] The multi-modal emotion recognition method based on label signal guidance proposed in the embodiment of the present invention first constructs a multi-modal emotion recognition model including a feature extraction module, a cross-modal interaction module, a label enhancement module, a dynamic fusion and classification module. By introducing label signals into the model explicitly and implicitly through the label enhancement module with a moving average mechanism and joint objective optimization, it can provide the key information required for emotion classification through label signals and further improve the expression ability of emotion features under the guiding role of label signals. The label signal embedding vector not only provides additional emotion category information for audio and text features, but also performs attention calculations with audio and text features respectively. The label embedding is used as a query to perform attention calculations with text features and audio features respectively, fully mining the emotion information in different modal data. And in the process of updating the label signal embedding vector, a moving average mechanism is used to control the update rate, greatly avoiding overfitting and information loss. At the same time, a joint objective optimization loss is designed, combining the attribution prediction consistency (APC) constraint and the cross-entropy loss function to optimize the consistency between the fused features and the emotion labels. The APC loss not only enhances the supervision role of label information in the training process by minimizing the difference between the similarity distribution between the fused features and the label vector and the model prediction distribution, but also enables the fused features to be more accurately aligned with the emotion categories.

[0089] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present invention. Therefore, the "in one embodiment" or "in an embodiment" that appears throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present invention, the order numbers of the above processes do not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention. The serial numbers of the embodiments of the present invention above are only for description and do not represent the advantages and disadvantages of the embodiments.

[0090] It should be noted that, in this text, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or device including that element.

[0091] In several embodiments provided by the present invention, it should be understood that the disclosed method can be implemented in other ways. The methods disclosed in several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments. The features disclosed in several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0092] As described above, the above is only the implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A multimodal emotion recognition method based on label signal guidance, characterized in that: include: Obtain a multimodal emotional data sample set, where each sample contains associated audio information and text information; A multimodal emotion recognition model is constructed, including a feature extraction module, a cross-modal interaction module, a label enhancement module, and a dynamic fusion and classification module; wherein the feature extraction module includes a first branch for extracting initial text features and a second branch for extracting initial audio features; the cross-modal interaction module is used to perform cross-modal attention calculation on the initial text features and the initial audio features to obtain a first text feature and a first audio feature; the label enhancement module is used to initialize the label information as a label embedding, and respectively enhance the first text feature and the first audio feature as a query vector, and respectively weightedly fuse the enhanced features with the respective corresponding initial features to obtain a final text feature and a final audio feature; the dynamic fusion and classification module is used to dynamically fuse the final audio feature and the final text feature using an expert gated fusion mechanism, and then output the predicted probability of each emotion category; Based on the emotion data sample, the multimodal emotion recognition model is iteratively trained using the designed joint objective optimization loss; wherein the joint objective optimization loss is obtained by combining the attribution prediction consistency APC loss and the cross entropy loss function; The emotion information to be identified is identified through the trained multimodal emotion recognition model to obtain the emotion prediction result.

2. The method according to claim 1, characterized in that In the first branch, Roberta is first used to extract the text features of the input text information. The text features are further captured through the Bi-LSTM layer to obtain the initial text features H t0 ; In the second branch, WavLM is used to extract the first feature of the input audio information, and the Mel spectrum of the audio information is extracted at the same time. The reshaped spectrum image is extracted by the pre-trained AlexNet to obtain the second feature, and the first feature and the second feature are added to obtain the initial audio feature H a0 .

3. The method according to claim 1, characterized in that The cross-modal interaction module uses one of the initial text feature and the initial audio feature as a query vector, and uses the other as a key and value to perform attention calculation, and obtains the first text feature H containing audio information. t1 and the first audio feature H containing text information a1 .

4. The method according to claim 1, characterized in that: In the tag enhancement module, the tag embedding is used as the query vector Q, and the first audio feature H a1 As key K a-t Sum value V a-t , perform attention calculation to obtain the second audio feature H a2 ; At the same time, the first text feature H t1 As key K t-a Sum value V t-a , perform attention calculation to obtain the second text feature H t2 ; And the final audio feature H is obtained by the following formula a And the final text feature H t : H a =sigmoid(H a2 )⊙H a0 +H a2 ; H t =sigmoid(H t2 )⊙H t0 +H t2 ; In the formula, ⊙ represents element-by-element multiplication; H a0 , H t0 are the initial audio features and initial text features respectively; sigmoid() is the activation function.

5. The method according to claim 4, characterized in that During model training, the label enhancement module updates label embeddings in the following way and uses a moving average mechanism to control the update rate: L t =αL t-1 +(1-α)L′ t-1 ; Where α is a hyperparameter that controls the update rate, L t-1 is the label embedding at the beginning of the t-1th training round, L' t-1 is the label embedding updated after the t-1th training round, L t is the label embedding at the beginning of the t-th training epoch; when t is 1, the label embedding is randomly initialized.

6. The method according to any one of claims 1 to 5, characterized in that: The dynamic fusion of the final audio features and the final text features includes: The final audio features and the final text features are connected into a combined feature vector, and the combined feature vector is passed through a fully connected layer to calculate the dynamic weights of the final audio features and the final text features: The final audio features and the final text features are weighted based on dynamic weights to obtain the final fusion feature representation; Send the fused feature representation into the multi-layer perceptron to obtain the emotion category score of the sample; The softmax function is used to normalize the emotion category score of each sample to obtain the probability that the sample belongs to each emotion category.

7. The method according to any one of claims 1 to 5, characterized in that: The joint objective optimization loss is expressed by the following formula: L(θ)=αL ce (θ)+βL apc (i); Among them, L apc (θ) is the APC loss, L ce (θ) is the cross entropy loss, L(θ) is the joint objective optimization loss, α and β are the balance weights of the cross entropy loss and APC loss, M is the total number of samples in the whole batch, N is the total number of emotion categories, and p i,j is the predicted probability that the i-th sample belongs to the j-th emotion category, q i,j is the similarity probability between the i-th sample and the j-th emotion category, is the JS divergence between the similarity distribution and the predicted probability distribution, y i,j is the true label.

Images