Robust multi-mode emotion recognition method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a robust multi-mode emotion recognition method and device, computer equipment, a computer readable storage medium and a computer program product, and relates to the technical field of natural language processing. The method comprises the following steps: acquiring a to-be-recognized corpus in a target scene; performing voice recognition processing on the to-be-recognized corpus to obtain a to-be-recognized text; inputting the to-be-recognized corpus and the to-be-recognized text into a pre-trained multi-modal emotion recognition model to obtain an emotion tag corresponding to the to-be-recognized corpus; the pre-trained multi-modal emotion recognition model is obtained by training after implicit feature reconstruction and feature alignment are carried out through a preset corpus; the preset corpus comprises clean corpora and corpora with noise; the implicit feature reconstruction is used for complementing damaged data of the noisy corpus relative to the clean corpus; the feature alignment is used for reducing the distribution difference of the feature space of the noisy corpus and the feature space of the clean corpus. By adopting the method, the emotion recognition accuracy of the voice can be improved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a robust multimodal emotion recognition method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Speech emotion recognition is a key task in intelligent speech assistance, which involves judging the speaker's emotional state, such as joy, anger, sadness, and fear, from speech signals. The most mainstream method at present is to extract features based on pre-trained models, and then deeply fuse the features of the two modalities of speech and text to obtain emotion recognition results. However, most of these methods only consider relatively ideal and static scenarios. In real application scenarios, multimodal signals are easily contaminated by various noises. For example, speech signals may be affected by silence or background noise, and transcribed text may also be disturbed by errors in automatic speech recognition, resulting in a serious decline in system performance. Therefore, current emotion recognition methods have the problem of low accuracy. Summary of the invention

[0003] Based on this, it is necessary to provide a robust multimodal emotion recognition method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of emotion recognition in speech in order to address the above technical problems.

[0004] In a first aspect, the present application provides a robust multimodal emotion recognition method, comprising:

[0005] Obtain the corpus to be recognized in the target scenario;

[0006] Performing speech recognition processing on the corpus to be recognized to obtain a text to be recognized;

[0007] The corpus to be recognized and the text to be recognized are input into a pre-trained multimodal emotion recognition model to obtain the emotion label corresponding to the corpus to be recognized; the pre-trained multimodal emotion recognition model is obtained by training after implicit feature reconstruction and feature alignment based on a preset corpus; the preset corpus includes clean corpus and noisy corpus; the implicit feature reconstruction is used to complete the damaged data of the noisy corpus relative to the clean corpus; the feature alignment is used to reduce the distribution difference of the feature space between the noisy corpus and the clean corpus.

[0008] In one embodiment, the pre-trained multimodal emotion recognition model includes a pre-trained speech encoder and a pre-trained text encoder, and the step of inputting the corpus to be recognized and the text to be recognized into the pre-trained multimodal emotion recognition model includes:

[0009] Inputting the corpus to be recognized into the pre-trained speech encoder to obtain primary speech features;

[0010] The text to be recognized is input into the pre-trained text encoder to obtain primary features of the text.

[0011] In one embodiment, the pre-trained multimodal emotion recognition model includes a pre-trained multimodal Transformer encoder and an emotion classifier, and the step of inputting the corpus to be recognized and the text to be recognized into the pre-trained multimodal emotion recognition model to obtain an emotion label corresponding to the corpus to be recognized includes:

[0012] Inputting the primary speech features and the primary text features into the pre-trained multimodal Transformer encoder respectively to obtain latent speech features and latent text features;

[0013] Performing average pooling and feature concatenation operations on the speech implicit features and the text implicit features in sequence to obtain multimodal high-level features;

[0014] The multimodal high-level features are input into a sentiment classifier to obtain sentiment labels corresponding to the corpus to be identified.

[0015] In one embodiment, the pre-trained multimodal emotion recognition model is obtained by the following method:

[0016] Performing feature extraction on the preset corpus to obtain primary features of the clean corpus and primary features of the noisy corpus;

[0017] Using a multimodal Transformer encoder to extract features from the primary features of the clean corpus and the primary features of the noisy corpus respectively, to obtain implicit features of the clean corpus and implicit features of the noisy corpus;

[0018] Performing decoding processing and reconstruction processing on the implicit features of the noisy corpus in sequence to obtain the reconstructed implicit features of the noisy corpus and a local feature reconstruction target function;

[0019] Respectively calculating the multimodal high-level features corresponding to the implicit features of the clean corpus and the reconstructed implicit features of the noisy corpus;

[0020] Performing feature alignment processing on the multimodal high-level features of the clean corpus and the multimodal high-level features of the noisy corpus to obtain a feature alignment objective function;

[0021] According to the local feature reconstruction objective function and the feature alignment objective function, the preset emotion recognition model is trained to obtain a pre-trained multimodal emotion recognition model.

[0022] In one embodiment, the training of a preset emotion recognition model according to the local feature reconstruction objective function and the feature alignment objective function to obtain a pre-trained multimodal emotion recognition model includes:

[0023] Constructing a sentiment classification task loss function between the output of the preset sentiment recognition model and the corresponding sentiment label;

[0024] Determine the overall training loss function according to the sum of the local feature reconstruction objective function, the feature alignment objective function and the sentiment classification task loss function;

[0025] With the goal of reducing the overall training loss function, the preset emotion recognition model is trained to obtain a pre-trained multimodal emotion recognition model.

[0026] In one embodiment, the clean corpus includes clean speech and clean text; before extracting features from the preset corpus to obtain primary features of the clean corpus and primary features of the noisy corpus, the method further includes:

[0027] The clean speech is subjected to random masking and noise addition processing in sequence to obtain noisy speech;

[0028] Performing random masking processing on the clean text to obtain noisy text;

[0029] A noisy corpus is obtained according to the noisy speech and the noisy text.

[0030] In a second aspect, the present application also provides a robust multimodal emotion recognition device, comprising:

[0031] Corpus acquisition module, used to acquire the corpus to be recognized in the target scenario;

[0032] A corpus processing module is used to perform speech recognition processing on the corpus to be recognized to obtain a text to be recognized;

[0033] A corpus classification module is used to input the corpus to be recognized and the text to be recognized into a pre-trained multimodal emotion recognition model to obtain the emotion label corresponding to the corpus to be recognized; the pre-trained multimodal emotion recognition model is obtained by training after implicit feature reconstruction and feature alignment based on a preset corpus; the preset corpus includes clean corpus and noisy corpus; the implicit feature reconstruction is used to complete the damaged data of the noisy corpus relative to the clean corpus; the feature alignment is used to reduce the distribution difference of the feature space between the noisy corpus and the clean corpus.

[0034] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0035] Obtain the corpus to be recognized in the target scenario;

[0036] Performing speech recognition processing on the corpus to be recognized to obtain a text to be recognized;

[0037] The corpus to be recognized and the text to be recognized are input into a pre-trained multimodal emotion recognition model to obtain the emotion label corresponding to the corpus to be recognized; the pre-trained multimodal emotion recognition model is obtained by training after implicit feature reconstruction and feature alignment based on a preset corpus; the preset corpus includes clean corpus and noisy corpus; the implicit feature reconstruction is used to complete the damaged data of the noisy corpus relative to the clean corpus; the feature alignment is used to reduce the distribution difference of the feature space between the noisy corpus and the clean corpus.

[0038] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0039] Obtain the corpus to be recognized in the target scenario;

[0040] Performing speech recognition processing on the corpus to be recognized to obtain a text to be recognized;

[0041] The corpus to be recognized and the text to be recognized are input into a pre-trained multimodal emotion recognition model to obtain the emotion label corresponding to the corpus to be recognized; the pre-trained multimodal emotion recognition model is obtained by training after implicit feature reconstruction and feature alignment based on a preset corpus; the preset corpus includes clean corpus and noisy corpus; the implicit feature reconstruction is used to complete the damaged data of the noisy corpus relative to the clean corpus; the feature alignment is used to reduce the distribution difference of the feature space between the noisy corpus and the clean corpus.

[0042] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0043] Obtain the corpus to be recognized in the target scenario;

[0044] Performing speech recognition processing on the corpus to be recognized to obtain a text to be recognized;

[0045] The corpus to be recognized and the text to be recognized are input into a pre-trained multimodal emotion recognition model to obtain the emotion label corresponding to the corpus to be recognized; the pre-trained multimodal emotion recognition model is obtained by training after implicit feature reconstruction and feature alignment based on a preset corpus; the preset corpus includes clean corpus and noisy corpus; the implicit feature reconstruction is used to complete the damaged data of the noisy corpus relative to the clean corpus; the feature alignment is used to reduce the distribution difference of the feature space between the noisy corpus and the clean corpus.

[0046] The above-mentioned robust multimodal emotion recognition method, device, computer equipment, computer-readable storage medium and computer program product obtain the corpus to be recognized in the target scene, and the target scene includes a real scene affected by noise. The corpus to be recognized is processed by speech recognition to obtain the text to be recognized. The corpus to be recognized and the text to be recognized are further input into the pre-trained multimodal emotion recognition model to obtain the emotion label corresponding to the corpus to be recognized; wherein the pre-trained multimodal emotion recognition model is obtained by training after implicit feature reconstruction and feature alignment based on a preset corpus. wherein the preset corpus includes clean corpus and noisy corpus, and the pre-trained multimodal emotion recognition model is obtained by training with noisy corpus and clean corpus together, so that the corpus to be recognized with noise can also be recognized by the model. wherein the implicit feature reconstruction is used to supplement the damaged data of the noisy corpus relative to the clean corpus, so that the pre-trained multimodal emotion recognition model has the ability to resist the lack of local features. Among them, feature alignment is used to reduce the distribution difference of the feature space between noisy corpus and clean corpus, which can improve the robustness of the pre-trained multimodal emotion recognition model to noise, thereby improving the emotion recognition accuracy of the pre-trained multimodal emotion recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0048] FIG1 is a schematic diagram of a process of a robust multimodal emotion recognition method in one embodiment;

[0049] FIG2 is a flow chart of the training steps of a multimodal emotion recognition model in one embodiment;

[0050] FIG3 is a schematic diagram of a flow chart of a step of determining an overall training loss in one embodiment;

[0051] FIG4 is a schematic diagram of a process flow of a robust multimodal emotion recognition method in another embodiment;

[0052] FIG5 is a structural block diagram of a robust multimodal emotion recognition device in one embodiment;

[0053] FIG. 6 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] In one embodiment, as shown in FIG1 , a robust multimodal emotion recognition method is provided. This embodiment is illustrated by applying the method to a terminal. It is understandable that the method can also be applied to a server, or to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0056] Step S102, obtaining the corpus to be recognized in the target scenario.

[0057] The target scenario may be a real application scenario where the speech signal may be polluted by various noises.

[0058] The corpus to be recognized may be a speech signal that needs to be subjected to emotion recognition, which is usually noisy speech.

[0059] Optionally, the system obtains the corpus to be recognized that needs to be used for emotion recognition in the target scene, which may be obtained by collecting the voice of the user in the target scene.

[0060] Step S104, performing speech recognition processing on the corpus to be recognized to obtain the text to be recognized.

[0061] The speech recognition processing may be ASR (Automatic Speech Recognition) technology, which is a technology that can convert speech audio into text.

[0062] Optionally, the system uses automatic speech recognition technology to perform speech recognition processing on the corpus to be recognized and convert it into corresponding text to be recognized.

[0063] Step S106, inputting the corpus to be recognized and the text to be recognized into a pre-trained multimodal emotion recognition model to obtain the emotion label corresponding to the corpus to be recognized.

[0064] The pre-trained multimodal emotion recognition model is trained by performing implicit feature reconstruction and feature alignment on a preset corpus; the preset corpus includes clean corpus and noisy corpus. Implicit feature reconstruction is used to complete the damaged data of the noisy corpus relative to the clean corpus. Feature alignment is used to reduce the distribution difference of the feature space between the noisy corpus and the clean corpus.

[0065] The sentiment label may be a sentiment classification label that is classified and annotated for each corpus file when constructing a preset corpus.

[0066] Optionally, the system inputs the corpus to be recognized and the text to be recognized into a pre-trained multimodal emotion recognition model. The pre-trained multimodal emotion recognition model extracts features from the corpus to be recognized and the text to be recognized respectively, and after a series of processing on the extracted features, outputs the emotion label corresponding to the corpus to be recognized, thereby determining the emotion recognition result of the corpus to be recognized.

[0067] In the above-mentioned robust multimodal emotion recognition method, by obtaining the corpus to be recognized in the target scene, the target scene includes a real scene affected by noise. The corpus to be recognized is processed by speech recognition to obtain the text to be recognized. The corpus to be recognized and the text to be recognized are further input into the pre-trained multimodal emotion recognition model to obtain the emotion label corresponding to the corpus to be recognized; wherein the pre-trained multimodal emotion recognition model is trained after implicit feature reconstruction and feature alignment through a preset corpus. wherein the preset corpus includes clean corpus and noisy corpus, and the pre-trained multimodal emotion recognition model is trained by jointly training the noisy corpus and the clean corpus, so that the corpus to be recognized with noise can also be recognized by the model. wherein the implicit feature reconstruction is used to supplement the data of the noisy corpus that is damaged relative to the clean corpus, so that the pre-trained multimodal emotion recognition model has the ability to resist the local feature loss. wherein the feature alignment is used to reduce the distribution difference of the feature space of the noisy corpus and the clean corpus, which can improve the robustness of the pre-trained multimodal emotion recognition model to noise, thereby improving the emotion recognition accuracy of the pre-trained multimodal emotion recognition model.

[0068] In an exemplary embodiment, the pre-trained multimodal emotion recognition model includes a pre-trained speech encoder and a pre-trained text encoder. Step S106 inputs the corpus to be recognized and the text to be recognized into the pre-trained multimodal emotion recognition model, including:

[0069] The corpus to be recognized is input into the pre-trained speech encoder to obtain the primary speech features; the text to be recognized is input into the pre-trained text encoder to obtain the primary text features.

[0070] Among them, the pre-trained speech encoder can be HuBERT (Hidden-Unit BERT, hidden unit BERT model), which adopts a Transformer-based encoder, which is pre-trained through self-supervised learning and is mainly used to process speech data.

[0071] Among them, the pre-trained text encoder can be RoBERTa (Reinforced Optimized BERT model), which is an improved version of the BERT model. It optimizes the performance of the model by using a larger training data set and longer training time. It is mainly used for text understanding and generation tasks.

[0072] Optionally, the system inputs the corpus to be recognized and the text to be recognized into a pre-trained speech encoder and a pre-trained text encoder respectively to obtain primary speech features and primary text features. Specifically, the corpus to be recognized and the text to be recognized are input into the HuBERT structure and the RoBERTa structure respectively to obtain primary speech features and primary text features for subsequent feature fusion.

[0073] In this embodiment, by performing primary feature extraction on the corpus to be recognized and the text to be recognized respectively for subsequent feature fusion, the complementary information between speech and text can be utilized to facilitate a more accurate model to understand the information of speech, thereby improving the accuracy of emotion recognition.

[0074] In an exemplary embodiment, the pre-trained multimodal emotion recognition model includes a pre-trained multimodal Transformer encoder and an emotion classifier. Step S106 inputs the corpus to be recognized and the text to be recognized into the pre-trained multimodal emotion recognition model to obtain the emotion label corresponding to the corpus to be recognized, including:

[0075] The primary features of speech and the primary features of text are respectively input into the pre-trained multimodal Transformer encoder to obtain the latent features of speech and the latent features of text; the latent features of speech and the latent features of text are successively subjected to average pooling and feature concatenation operations to obtain multimodal high-level features; the multimodal high-level features are input into the pre-trained sentiment classifier to obtain the sentiment labels corresponding to the corpus to be identified.

[0076] Among them, the implicit features are output by the pre-trained multimodal Transformer encoder after jointly modeling the intra-modal and inter-modal interaction relationships.

[0077] Among them, average pooling is a downsampling technique commonly used in convolutional neural networks (CNNs). It reduces the size of the feature map by calculating the average value of each local area in the input feature map. This method can effectively preserve the spatial information of the features while reducing the amount of calculation and the risk of overfitting. Average pooling is usually used after the convolution layer to reduce the dimension of the feature map.

[0078] Feature splicing is an operation that connects multiple feature vectors in a certain dimension. In deep learning, feature splicing is often used to combine features from different layers or different models for subsequent processing or classification. Through splicing, the advantages of different features can be fully utilized to improve the performance of the model. Feature splicing is usually performed before the fully connected layer to form a richer feature representation.

[0079] Optionally, the system inputs the primary features of speech and the primary features of text into the trained multimodal Transformer encoder to obtain the latent features of speech and the latent features of text. For example, the latent features of speech and the latent features of text are and , the specific steps to calculate the speech latent features and text latent features are: first perform Q, K, V projection on the speech latent features and text latent features respectively:

[0080] , ,

[0081] , ,

[0082] in, , , as well as , , All are learnable parameters. Then, the intra-modal relationship of the Q, K, and V matrices obtained after projection is modeled:

[0083] ,

[0084] in, is the square root of the K dimension, It is the intra-modal relationship after the projection of the primary features of speech. It is the intra-model relationship after the projection of the primary features of the text. It refers to normalization processing.

[0085] Model the inter-modal relationship of the Q, K, and V matrices obtained after projection:

[0086] ,

[0087] in, is the intermodal relationship between the primary features of speech and the primary features of text after projection, It is the inter-modal relationship between the primary features of text and the primary features of speech after projection.

[0088] The implicit features of speech and text are calculated separately, and the corresponding formulas are as follows:

[0089]

[0090]

[0091] in, represents the implicit features of speech, Represents the implicit features of the text, Concat represents the concatenation operation along the dimension direction, Linear represents the linear layer, and is a learnable parameter, and The feature dimension is projected to and Consistent.

[0092] The system performs average pooling and feature concatenation operations on the voice implicit features and text implicit features in turn to obtain multimodal high-level features. The corresponding multimodal high-level feature calculation formula is as follows:

[0093]

[0094] Among them, MeanPool represents the average pooling operation along the time direction. Represents multimodal high-level features.

[0095] After calculating the multimodal high-level features, the system inputs the multimodal high-level features into the sentiment classifier to obtain the sentiment labels corresponding to the corpus to be identified.

[0096] In this embodiment, by combining the implicit features of speech and text, the emotion of the corpus to be recognized can be understood more comprehensively. Multimodal learning can usually improve the accuracy of emotion recognition because it utilizes different types of data sources and reduces the deviation that may be caused by a single modality. Speech or text may contain noise or omissions. By combining multimodal features, the robustness of the system to these problems can also be enhanced.

[0097] In an exemplary embodiment, as shown in FIG2 , the pre-trained multimodal emotion recognition model described in the above embodiment is obtained by the following method:

[0098] Step S202: extract features from the preset corpus to obtain primary features of the clean corpus and primary features of the noisy corpus.

[0099] The preset corpus may be constructed in advance for a plurality of clean voices that have been emotion-labeled and classified, wherein the clean corpus includes clean voices and clean texts.

[0100] Optionally, the system performs data enhancement processing on the clean speech in the preset corpus to obtain noisy corpus, which also includes noisy speech and noisy text, and further performs feature extraction on the clean speech, noisy speech, clean text and noisy text to obtain primary features corresponding to the clean speech, noisy speech, clean text and noisy text respectively.

[0101] Step S204: Use a multimodal Transformer encoder to extract features from the primary features of the clean corpus and the primary features of the noisy corpus respectively, to obtain implicit features of the clean corpus and implicit features of the noisy corpus.

[0102] Optionally, the system uses a multimodal Transformer encoder to extract features from the primary features of the clean corpus and the primary features of the noisy corpus respectively to obtain implicit features of the clean corpus and implicit features of the noisy corpus. The extraction method is the same as the calculation method of the implicit features of the corpus to be identified in the above embodiment.

[0103] Step S206, decoding and reconstruction are performed on the implicit features of the noisy corpus in sequence to obtain the reconstructed implicit features of the noisy corpus and the local feature reconstruction target function.

[0104] Optionally, the system decodes the implicit features of the noisy corpus, and the corresponding formula is as follows:

[0105] The noisy latent features are mapped to the same dimension as the input features through the linear layer decoder:

[0106] ,

[0107] in, represents the reconstructed implicit features of noisy speech, represents the reconstructed implicit features of the noisy text, and are learnable parameters.

[0108] The system then reconstructs the implicit features of the noisy corpus to obtain the local feature reconstruction objective function. The corresponding formula is as follows:

[0109]

[0110] Among them, L1 represents the L1 norm loss function, represents the loss of local reconstruction, represents the primary features of noisy speech, Represents the primary features of noisy text, represents the masking matrix of the speech corpus, Represents the masking matrix for noisy text.

[0111] Step S208 , respectively calculating the multimodal high-level features corresponding to the implicit features of the clean corpus and the reconstructed implicit features of the noisy corpus.

[0112] Optionally, the system calculates the multimodal high-level features obtained by averaging and concatenating the implicit features of clean speech and clean text, and calculates the multimodal high-level features obtained by averaging and concatenating the implicit features of noisy speech and noisy text, and the corresponding formula is the same as that of the above embodiment. The multimodal Transformer encoders used by the noisy corpus and the clean corpus respectively use the same weights.

[0113] Step S210 , performing feature alignment processing on the multimodal high-level features of the clean corpus and the multimodal high-level features of the noisy corpus to obtain a feature alignment objective function.

[0114] Optionally, the system performs feature alignment processing on the multimodal high-level features of the clean corpus and the multimodal high-level features of the noisy corpus to obtain a feature alignment objective function. The corresponding formula is as follows:

[0115]

[0116] in, Represents the multimodal high-level features of noisy corpus, Represents the multimodal high-level features of clean corpus. JSD stands for Jensen-Shannon divergence, which can measure the degree of difference between two probability distributions. represents the feature alignment loss.

[0117] Step S212: training the preset emotion recognition model according to the local feature reconstruction objective function and the feature alignment objective function to obtain a pre-trained multimodal emotion recognition model.

[0118] Optionally, the system reconstructs the objective function and the feature alignment objective function based on the local features, takes the minimum loss value as the goal, and trains the preset emotion recognition model to obtain a pre-trained multimodal emotion recognition model.

[0119] In this embodiment, through a preset corpus, after a series of feature extraction processes, the feature reconstruction of the noisy corpus can complete the damaged data of the noisy corpus relative to the clean corpus, so that the pre-trained multimodal emotion recognition model has the ability to resist the lack of local features. Then, the implicit features of the noisy corpus and the clean corpus are aligned to reduce the distribution difference of the feature space of the noisy corpus and the clean corpus, which can improve the robustness of the multimodal emotion recognition model to noise. Finally, according to the local feature reconstruction objective function and the feature alignment objective function, the preset emotion recognition model is trained with the minimum loss value as the goal to obtain a pre-trained multimodal emotion recognition model, thereby obtaining a more robust multimodal emotion recognition model, further improving the emotion recognition accuracy of the system.

[0120] In an exemplary embodiment, as shown in FIG3 , step S212 trains a preset emotion recognition model according to a local feature reconstruction objective function and a feature alignment objective function to obtain a pre-trained multimodal emotion recognition model, including the following steps S302 to S306 . Among them:

[0121] Step S302, constructing a sentiment classification task loss function between the output of the preset sentiment recognition model and the corresponding sentiment label.

[0122] Among them, the output of the preset emotion recognition model can be the classification result output after the multimodal high-level features of the noisy corpus after feature alignment are input into the classifier of the preset emotion recognition model.

[0123] Optionally, the system constructs a sentiment classification task loss function between the output of a preset sentiment recognition model and the actual sentiment label corresponding to the noisy corpus, and the corresponding formula is:

[0124]

[0125] in, is the loss for the sentiment classification task, is the output of the preset emotion recognition model, is the actual sentiment label corresponding to the noisy corpus.

[0126] Step S304, determining the overall training loss function according to the sum of the local feature reconstruction objective function, the feature alignment objective function and the sentiment classification task loss function.

[0127] Optionally, the system determines the overall training loss function based on the sum of the local feature reconstruction objective function, the feature alignment objective function, and the sentiment classification task loss function. The corresponding formula is:

[0128]

[0129] in, is the overall training loss, is the loss function for the sentiment classification task, Reconstruct the objective function for local features, is the feature alignment objective function, and To control the hyperparameters of the adversarial training loss weight, the default setting is .

[0130] Step S306, with the goal of reducing the overall training loss function, the preset emotion recognition model is trained to obtain a pre-trained multimodal emotion recognition model.

[0131] Optionally, the system trains a preset emotion recognition model with the goal of reducing the overall training loss function until the training end condition is met, thereby obtaining a pre-trained multimodal emotion recognition model, wherein the training end condition may be that the overall training loss function is lower than a preset threshold, or that the training has reached a preset number of times.

[0132] In this embodiment, the loss function of the overall training is obtained by comprehensively considering the objective function under different feature processing conditions, so that the model performance can be more comprehensively optimized during the training process and the accuracy of emotion recognition can be improved.

[0133] In an exemplary embodiment, the clean corpus includes clean speech and clean text; before step S202 extracts features from the preset corpus to obtain primary features of the clean corpus and primary features of the noisy corpus, the method further includes:

[0134] The clean speech is subjected to random masking and noise addition processing in turn to obtain noisy speech; the clean text is subjected to random masking processing to obtain noisy text; and the noisy corpus is obtained based on the noisy speech and noisy text.

[0135] Among them, random masking can be a data augmentation method, which is usually used to process sequence data (such as text, audio, etc.) or image data. The basic idea is to randomly select a part of the elements in the input data for masking (i.e., replace them with specific values, such as zero or other placeholders) to simulate the situation of missing data.

[0136] Among them, noise addition processing can be a data enhancement technique that improves the robustness of the model by adding random noise to the input data. The noise can be Gaussian noise, salt and pepper noise, etc., depending on the application scenario.

[0137] Optionally, the system performs random masking and noise addition processing on the clean speech in sequence to obtain noisy speech. For example, the system randomly selects a noise from a given non-stationary noise database, and randomly selects the signal-to-noise ratio in the range of 0~20dB for additive noise aliasing to simulate the real environment. After the original speech waveform is divided into frames with a frame length of 25ms and a frame shift of 10ms, 20% of the speech frames are randomly set to zero to simulate the situation of missing data, and the time sequence position of the zeroing is recorded through the masking matrix to obtain noisy speech. In addition, the system performs random masking processing on the clean text to obtain noisy text. For example, the system also performs a masking operation on 20% of the characters in the text sequence of the clean text to simulate the unrecognizable [UNK] character, and records it through the masking matrix. The system further obtains a noisy corpus based on the noisy speech and noisy text.

[0138] In this embodiment, by randomly selecting noise from a non-stationary noise database and aliasing it at different signal-to-noise ratios, the speech features in various noise environments can be learned in subsequent model training. This training method enables the model to better recognize and understand speech content when facing background noise in a real environment. And 20% of the speech frames are set to zero to simulate the situation of missing data, so that the model can still maintain a high recognition accuracy when processing incomplete information. This method enhances the model's adaptability to missing data.

[0139] In an exemplary embodiment, as shown in FIG4 , another robust multimodal emotion recognition method is provided, and the specific steps include:

[0140] Step 1: Dynamic enhancement of multimodal noisy data.

[0141] For example, during the training phase, the following two steps of data enhancement are performed on the given clean multimodal data to obtain the noisy data: (1) Background noise simulation: Given a non-stationary noise database , randomly select a noise from it, and randomly select the signal-to-noise ratio in the range of 0~20dB to perform additive noise aliasing to simulate the real environment. (2) Data missing simulation: The original speech waveform is divided into frames with a frame length of 25ms and a frame shift of 10ms. 20% of the speech frames are randomly set to zero to simulate data missing. Record the timing position of zeroing; 20% of the characters in the text sequence are also masked to simulate the unrecognizable [UNK] characters, and the masking matrix Record.

[0142] Step 2: Primary feature extraction and multimodal advanced feature modeling based on pre-trained models

[0143] Step 2.1, primary feature extraction. For example, the speech and text pre-training models in the present invention use HuBERT and RoBERTa structures respectively, and fix the weights of the pre-trained encoder as the multimodal primary feature extraction module to obtain the primary features of clean speech and text. and , primary features of noisy speech and text and , used for subsequent feature fusion.

[0144] Step 2.1, multimodal high-level feature modeling. For example, the multimodal Transformer encoder in the present invention jointly models the intra-modal and inter-modal interaction relationships and outputs noisy speech and text implicit representations. and Take this as an example, the specific calculation steps are as follows:

[0145] Q,K,V projection:

[0146] , ,

[0147] , ,

[0148] in, , , as well as , , All are learnable parameters. (2) Intra-modal relationship modeling:

[0149] ,

[0150] Modeling inter-modal relationships:

[0151] ,

[0152] Implicit feature calculation:

[0153]

[0154]

[0155] Among them, Concat represents the concatenation operation along the dimension direction, Linear represents the linear layer, and is a learnable parameter, and The feature dimension is projected to and Consistent.

[0156] Multimodal advanced feature computation:

[0157]

[0158] Among them, MeanPool represents the average pooling operation along the time direction. At the same time, clean speech and text and Input the multimodal Transformer encoder with the same structure and parameters to obtain the corresponding hidden features and , and multimodal high-level features .

[0159] Step 3: Two-stage feature reconstruction. Exemplarily, the present invention performs feature reconstruction from both local and global perspectives based on the clean and noisy features extracted in step 5.2 to improve the robustness of the model in the inference stage.

[0160] Step 3.1, local feature decoding and reconstruction. For example, the implicit features of noisy speech and text and Encoding sufficient multimodal context information, the present invention proposes to guide implicit features to complete the information of noisy data relative to the damaged clean data to encourage the model to have the ability to resist local feature loss. The specific calculation steps are as follows:

[0161] The noisy latent features are mapped to the same dimension as the input features through the linear layer decoder:

[0162] ,

[0163] Local feature reconstruction objective function:

[0164]

[0165] Among them, L1 represents the L1 norm loss function.

[0166] Step 3.2, global feature alignment. For example, the high-level multimodal features of noisy and clean data highly abstract the emotional semantics of the input features, and can directly enhance the model's robustness to noise by reducing the distribution difference between the two in the feature space. The objective function of global feature alignment is as follows:

[0167]

[0168] Among them, JSD stands for Jensen-Shannon divergence, which can measure the degree of difference between two probability distributions.

[0169] Step 4: Model training objectives.

[0170] For example, the output of the sentiment classifier is recorded as , and the corresponding labels are , then the loss function of the sentiment classification task is defined as follows:

[0171]

[0172] According to the above analysis, the overall training loss function for the robust multimodal emotion recognition method can be expressed as:

[0173]

[0174] in and To control the hyperparameters of the adversarial training loss weight, the default setting is In the training phase, the parameters of the pre-trained encoders for speech and text are fixed, and the parameters of the remaining trainable modules are updated through the stochastic gradient descent algorithm. In the application reasoning phase, only the forward calculation steps except deconstruction and reconstruction of noisy speech and noisy text as shown in Figure 4 need to be performed.

[0175] In this embodiment, a method for dynamic enhancement of multimodal noisy data is proposed, which dynamically generates noisy data pairs for a given multimodal sentiment corpus. On the one hand, it expands the amount of data and alleviates the problem of model overfitting; on the other hand, it provides paired clean and noisy paired data for subsequent feature reconstruction. A two-stage feature reconstruction method is proposed, which guides each modality to use context information to restore missing features through local feature reconstruction, and guides the model to reduce the emotional semantic differences between clean and noisy high-level multimodal representations through global feature alignment, combining the complementary advantages of the two to improve the robustness of the model against noisy data.

[0176] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0177] Based on the same inventive concept, the embodiment of the present application also provides a robust multimodal emotion recognition method device for implementing the above-mentioned robust multimodal emotion recognition method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more robust multimodal emotion recognition method device embodiments provided below can refer to the limitations of the robust multimodal emotion recognition method above, and will not be repeated here.

[0178] In an exemplary embodiment, as shown in FIG5 , a robust multimodal emotion recognition method and apparatus 500 is provided, comprising: a corpus acquisition module 502 , a corpus processing module 504 , and a corpus classification module 506 , wherein:

[0179] The corpus acquisition module 502 is used to acquire the corpus to be recognized in the target scenario.

[0180] The corpus processing module 504 is used to perform speech recognition processing on the corpus to be recognized to obtain the text to be recognized.

[0181] The corpus classification module 506 is used to input the corpus to be recognized and the text to be recognized into the pre-trained multimodal emotion recognition model to obtain the emotion label corresponding to the corpus to be recognized; the pre-trained multimodal emotion recognition model is obtained by training after implicit feature reconstruction and feature alignment based on a preset corpus; the preset corpus includes clean corpus and noisy corpus; implicit feature reconstruction is used to complete the damaged data of the noisy corpus relative to the clean corpus; feature alignment is used to reduce the distribution difference of the feature space between the noisy corpus and the clean corpus.

[0182] Furthermore, in one embodiment, the corpus classification module 506 is further configured to input the corpus to be recognized into a pre-trained speech encoder to obtain primary speech features; and input the text to be recognized into a pre-trained text encoder to obtain primary text features.

[0183] Furthermore, in one embodiment, the corpus classification module 506 is also used to input the primary features of speech and the primary features of text into a pre-trained multimodal Transformer encoder respectively to obtain latent features of speech and latent features of text; perform average pooling and feature concatenation operations on the latent features of speech and text in turn to obtain multimodal high-level features; and input the multimodal high-level features into a pre-trained sentiment classifier to obtain sentiment labels corresponding to the corpus to be identified.

[0184] Furthermore, in one embodiment, the robust multimodal emotion recognition method device 500 also includes a model training module, which is used to perform feature extraction on a preset corpus to obtain primary features of a clean corpus and primary features of a noisy corpus; use a multimodal Transformer encoder to perform feature extraction on the primary features of the clean corpus and the primary features of the noisy corpus respectively to obtain implicit features of the clean corpus and implicit features of the noisy corpus; perform decoding processing and reconstruction processing on the implicit features of the noisy corpus in turn to obtain the reconstructed implicit features of the noisy corpus and the local feature reconstruction objective function; calculate the multimodal high-level features corresponding to the implicit features of the clean corpus and the reconstructed implicit features of the noisy corpus respectively; perform feature alignment processing on the multimodal high-level features of the clean corpus and the multimodal high-level features of the noisy corpus to obtain the feature alignment objective function; according to the local feature reconstruction objective function and the feature alignment objective function, train the preset emotion recognition model to obtain a pre-trained multimodal emotion recognition model.

[0185] Furthermore, in one embodiment, the model training module is also used to construct a sentiment classification task loss function between the output of a preset sentiment recognition model and the corresponding sentiment label; determine the overall training loss function based on the sum of the local feature reconstruction objective function, the feature alignment objective function and the sentiment classification task loss function; and train the preset sentiment recognition model with the goal of reducing the overall training loss function to obtain a pre-trained multimodal sentiment recognition model.

[0186] Furthermore, in one embodiment, the model training module is also used to perform random masking and noise addition processing on clean speech in sequence to obtain noisy speech; perform random masking processing on clean text to obtain noisy text; and obtain noisy corpus based on the noisy speech and noisy text.

[0187] Each module in the above-mentioned robust multimodal emotion recognition device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0188] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG6. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as corpus to be recognized, text to be recognized, and emotion tags. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a robust multimodal emotion recognition method is implemented.

[0189] Those skilled in the art will understand that the structure shown in FIG. 6 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0190] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0191] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0192] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0193] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0194] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0195] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0196] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A robust multimodal emotion recognition method, characterized in that: The method comprises: Obtain the corpus to be recognized in the target scenario; Performing speech recognition processing on the corpus to be recognized to obtain a text to be recognized; The corpus to be recognized and the text to be recognized are input into a pre-trained multimodal emotion recognition model to obtain the emotion label corresponding to the corpus to be recognized; the pre-trained multimodal emotion recognition model is obtained by training after implicit feature reconstruction and feature alignment based on a preset corpus; the preset corpus includes clean corpus and noisy corpus; the implicit feature reconstruction is used to complete the damaged data of the noisy corpus relative to the clean corpus; the feature alignment is used to reduce the distribution difference of the feature space between the noisy corpus and the clean corpus.

2. The method according to claim 1, characterized in that: The pre-trained multimodal emotion recognition model includes a pre-trained speech encoder and a pre-trained text encoder, and the step of inputting the corpus to be recognized and the text to be recognized into the pre-trained multimodal emotion recognition model includes: Inputting the corpus to be recognized into the pre-trained speech encoder to obtain primary speech features; The text to be recognized is input into the pre-trained text encoder to obtain primary features of the text.

3. The method according to claim 2, characterized in that The pre-trained multimodal emotion recognition model includes a pre-trained multimodal Transformer encoder and an emotion classifier. The step of inputting the corpus to be recognized and the text to be recognized into the pre-trained multimodal emotion recognition model to obtain an emotion label corresponding to the corpus to be recognized includes: Inputting the primary speech features and the primary text features into the pre-trained multimodal Transformer encoder respectively to obtain latent speech features and latent text features; Performing average pooling and feature concatenation operations on the speech implicit features and the text implicit features in sequence to obtain multimodal high-level features; The multimodal high-level features are input into a sentiment classifier to obtain sentiment labels corresponding to the corpus to be identified.

4. The method according to claim 1, characterized in that: The pre-trained multimodal emotion recognition model is obtained by the following method: Performing feature extraction on the preset corpus to obtain primary features of the clean corpus and primary features of the noisy corpus; Using a multimodal Transformer encoder to extract features from the primary features of the clean corpus and the primary features of the noisy corpus respectively, to obtain implicit features of the clean corpus and implicit features of the noisy corpus; Performing decoding processing and reconstruction processing on the implicit features of the noisy corpus in sequence to obtain the reconstructed implicit features of the noisy corpus and a local feature reconstruction target function; Respectively calculating the multimodal high-level features corresponding to the implicit features of the clean corpus and the reconstructed implicit features of the noisy corpus; Performing feature alignment processing on the multimodal high-level features of the clean corpus and the multimodal high-level features of the noisy corpus to obtain a feature alignment objective function; According to the local feature reconstruction objective function and the feature alignment objective function, the preset emotion recognition model is trained to obtain a pre-trained multimodal emotion recognition model.

5. The method according to claim 4, characterized in that The method of training a preset emotion recognition model according to the local feature reconstruction objective function and the feature alignment objective function to obtain a pre-trained multimodal emotion recognition model includes: Constructing a sentiment classification task loss function between the output of the preset sentiment recognition model and the corresponding sentiment label; Determine the overall training loss function according to the sum of the local feature reconstruction objective function, the feature alignment objective function and the sentiment classification task loss function; With the goal of reducing the overall training loss function, the preset emotion recognition model is trained to obtain a pre-trained multimodal emotion recognition model.

6. The method according to claim 4, characterized in that The clean corpus includes clean speech and clean text; before extracting features from the preset corpus to obtain primary features of the clean corpus and primary features of the noisy corpus, the method further includes: The clean speech is subjected to random masking and noise addition processing in sequence to obtain noisy speech; Performing random masking processing on the clean text to obtain noisy text; A noisy corpus is obtained according to the noisy speech and the noisy text.

7. A robust multimodal emotion recognition device, characterized in that: The device comprises: Corpus acquisition module, used to acquire the corpus to be recognized in the target scenario; A corpus processing module is used to perform speech recognition processing on the corpus to be recognized to obtain a text to be recognized; A corpus classification module is used to input the corpus to be recognized and the text to be recognized into a pre-trained multimodal emotion recognition model to obtain the emotion label corresponding to the corpus to be recognized; the pre-trained multimodal emotion recognition model is obtained by training after implicit feature reconstruction and feature alignment based on a preset corpus; the preset corpus includes clean corpus and noisy corpus; the implicit feature reconstruction is used to complete the damaged data of the noisy corpus relative to the clean corpus; the feature alignment is used to reduce the distribution difference of the feature space between the noisy corpus and the clean corpus.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images