Multi-modal emotion recognition method and device for modal missing and out-of-distribution data

Abstract

The invention discloses a multi-modal emotion recognition method and device oriented to modal missing and out-of-distribution data, and the method comprises the steps: firstly obtaining user text, audio and visual data, extracting each modal feature through a feature extraction module, then extracting a time sequence feature corresponding to each modal feature subjected to word-level alignment, and carrying out the feature extraction through a feature extraction module; the method comprises the following steps of: splicing time sequence features of each mode in a time dimension, inputting the spliced time sequence features into a multi-mode composite module to obtain composite features, then processing the composite features to obtain multi-mode joint representation, and finally inputting the multi-mode joint representation into a classification module to obtain an identification result. According to the technical scheme, model-specific self-distillation is used for coping with a mode missing challenge, model-irrelevant causal inference is used for coping with an out-of-distribution data challenge, the problems of mode missing and out-of-distribution data can be solved, and the emotion recognition accuracy can be improved.

Description

Technical Field

[0001] The present application belongs to the field of emotion recognition technology, and in particular relates to a multimodal emotion recognition method and device for modality-missing and out-of-distribution data. Background Art

[0002] People express emotions in a variety of ways. In addition to the most basic language, they often use various other modalities, such as pitch, facial expressions, and body movements. Using the information contained in multiple modalities to analyze the emotions expressed by a person is multimodal emotion recognition.

[0003] In the current field of multimodal sentiment analysis, researchers often explore effective methods for integrating multiple modalities based on word-level aligned multimodal sequences. However, these efforts overlook the two critical challenges faced simultaneously in practical applications: modality loss and out-of-distribution data. Therefore, effectively addressing these two currently challenging issues in multimodal sentiment analysis remains of great theoretical and practical value.

[0004] Currently, with the continuous maturity and development of various neural network models and computing resources, deep learning methods have achieved excellent results in various fields. In recent years, a large number of methods for multimodal emotion recognition have emerged. In addition to the methods mentioned above that focus on multimodal fusion, there has also been some work targeting random modal feature omissions or random modal missingness. These works can be divided into two categories: the first category uses modality translation methods, using information from existing modalities to attempt to translate the missing modality, and finally uses the completed full multimodal sequence for multimodal emotion recognition; the second category uses the reconstruction of the location of missing modal features to guide the front encoder to learn the semantic information of the missing modal feature location, thereby effectively completing multimodal emotion recognition without the need to complete the full multimodal sequence. In addition, for out-of-distribution data, a number of works have emerged that improve its generalization ability through causal inference or probabilistic modeling.

[0005] However, these methods either only address the problem of missing modalities or only address the problem of out-of-distribution data. When both occur at the same time, these methods often cannot continue to be competent for emotion recognition. Summary of the Invention

[0006] The purpose of this application is to provide a multimodal emotion recognition method and device for modal missing and out-of-distribution data, to overcome the problem that previous work only solves one aspect of modal missing or out-of-distribution data, and to further improve the final emotion recognition performance.

[0007] In order to achieve the above objectives, the technical solutions of this application are as follows: A multimodal emotion recognition method for modality-missing and out-of-distribution data performs emotion recognition by constructing and training a multimodal emotion recognition network model. The multimodal emotion recognition network model includes a feature extraction module, a word-level self-aligned attention module, a multimodal composite module, a joint representation module, and a classification module. The multimodal emotion recognition method for modality-missing and out-of-distribution data includes: Obtain user text, audio, and visual data, and extract features of each modality through the feature extraction module; A word-level self-aligned attention module is used to adaptively align the features of each modality at the word level, and then the temporal features corresponding to the aligned modal features are extracted; The temporal features of each modality are spliced ​​in the time dimension, and then the spliced ​​multimodal sequence is input into the multimodal composite module to obtain the composite features; In the joint representation module, each modality of the composite feature is extracted separately and an attention mechanism is applied to obtain a discourse-level representation of each modality. The discourse-level representations of each modality are then concatenated and passed through a multi-layer perceptron to obtain a multimodal joint representation. Input the multimodal joint representation into the classification module to obtain the recognition result; Wherein, the multimodal composite module includes N Multimodal composite layers, each multimodal composite layer is connected in sequence. i A multimodal composite layer performs the following operations: After the input features are normalized in the first layer, the first features are obtained; The first feature is input into the multimodal hybrid attention module to obtain the second feature; After the first feature is added to the second feature, the third feature is obtained after the second layer normalization; The third feature is added to the third feature after passing through the feedforward network to obtain the output feature of the multimodal composite layer; The multimodal hybrid attention module performs the following operations: For the input features, calculate the self-attention matrix; Add the self-attention matrix to each unimodal and bimodal mask matrix to generate each unimodal attention matrix and bimodal attention matrix; The self-attention matrix, each unimodal attention matrix and bimodal attention matrix are respectively softmax After the operation, it is multiplied by the value vector corresponding to the input feature to obtain the corresponding attention representation; The attention representation is mapped through a linear layer, and then the effective parts of the mapped features are extracted separately and spliced ​​in the feature dimension to obtain the spliced ​​features of the three modalities of text, audio and vision; After the three modal features are spliced ​​together, they are reduced in dimension through a linear layer, then spliced ​​together in the time dimension, and then passed through a linear layer to obtain the final output.

[0008] Furthermore, the classification module performs the following operations: A multi-layer perceptron is used to encode the prior features of each modality to obtain the coded features of each modality; The encoding features of each modality are concatenated with the multimodal joint representation and then input into the fully connected layer to obtain the preliminary classification results; The preliminary classification result is multiplied by the prior probability and summed to obtain the recognition result after label deviation correction.

[0009] Furthermore, the multimodal emotion recognition method for modality-missing and out-of-distribution data includes: The recognition result of the counterfactual text is subtracted from the text recognition result output by the classification module to obtain the final recognition result.

[0010] Furthermore, when training the multimodal emotion recognition network model, the input training samples include incomplete data samples and complete data samples, and the following loss is calculated during training: In the case of incomplete data sample input, the composite features are reconstructed to obtain the corresponding reconstructed features, and then the regression loss function is used to calculate the reconstruction loss between the reconstructed features and the features of each modality extracted by the feature extraction module; In the case of complete data samples and incomplete data samples, the self-attention matrix output by the last multimodal composite layer is obtained respectively, and then the KL divergence loss of the self-attention matrix is ​​calculated; The cosine similarity loss of the multimodal joint representation is calculated when both complete and incomplete data samples are input.

[0011] Furthermore, when training the multimodal emotion recognition network model, a two-stage loss calculation and model parameter update process is included, wherein: In the first stage, the complete data sample is input, the cross entropy loss corresponding to the recognition result output by the classification module is calculated, back propagation is performed, and the model parameters are updated; In the second stage, incomplete data samples are input, and the cross entropy loss corresponding to the recognition result output by the classification module is calculated. The cross entropy loss is combined with the reconstruction loss, KL divergence loss, and cosine similarity loss to form a joint loss, and back propagation is performed to update the model parameters.

[0012] The present application also proposes a multimodal emotion recognition device for modality-missing and out-of-distribution data, comprising a processor and a memory storing a plurality of computer instructions, wherein the computer instructions implement the steps of the above method when executed by the processor.

[0013] This application proposes a multimodal emotion recognition method and apparatus for modality-missing and out-of-distribution data. This method uses model-specific self-distillation to address the challenge of modality-missing and model-independent causal inference to address the challenge of out-of-distribution data. The two methods are organically integrated to address both challenges simultaneously. This technical solution not only overcomes the problems of modality-missing and out-of-distribution data, but also improves the accuracy of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a flowchart of the multimodal emotion recognition method for modality-missing and out-of-distribution data in this application.

[0015] Figure 2 This is a structural diagram of an embodiment of the multimodal emotion recognition network model of this application.

[0016] Figure 3 This is a structural diagram of the multimodal composite module of an embodiment of the present application.

[0017] Figure 4 This is a structural diagram of another embodiment of the multimodal emotion recognition network model of this application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0019] Example 1: Figure 1 As shown, a multimodal emotion recognition method for modality-missing and out-of-distribution data is proposed, which performs emotion recognition by constructing and training a multimodal emotion recognition network model. The multimodal emotion recognition method for modality-missing and out-of-distribution data in this embodiment includes: Step S1: Obtain user text, audio and visual data, and extract features of each modality through a feature extraction module.

[0020] The multimodal emotion recognition network model constructed in this embodiment is as follows Figure 2 As shown in the figure, it includes feature extraction module, word-level self-aligned attention module (WSAM), multimodal composite module (MCT), joint representation module and classification module.

[0021] After the various text, audio, and video data published by users on the platform pass through the feature extraction module, the corresponding multimodal features can be obtained, which can be expressed as: , , ,in and Represent the sequence length and feature dimension of each modality respectively.

[0022] For complete data, it is recorded as , for incomplete data, it is recorded as ,in Indicates mode loss. For simplicity, the following uses When expressed, it means that both complete and incomplete data are processed in the same way.

[0023] It should be noted that the feature extraction module uses different neural network structures for different modalities, all of which use pre-trained neural networks for feature extraction. For example, text features are extracted by the BERT pre-trained model, audio features are extracted by COVAREP, and video features are extracted by Facet. The feature extraction module also includes a one-dimensional convolutional network, which maps each modal sequence into the same feature space. The final feature representation of each modality is: ; in, is the common feature dimension of the three modalities after mapping. Indicates each modality, namely text, audio and visual, with separate subscripts 、 、 They represent the corresponding single modes respectively.

[0024] Step S2: Adopt the word-level self-alignment attention module to adaptively align the modal features at the word level, and then extract the temporal features corresponding to the modal features that have been aligned at the word level.

[0025] In order to minimize the computational complexity and training time of the network, this embodiment proposes a word-level self-aligned attention module (WSAM) for adaptive word-level alignment of audio, video, and text modal features.

[0026] In visual mode For example, the text mode After linear transformation, the query vector (Query) is obtained and the visual modality After linear transformation, the key vector (Key) and value vector (Value) are obtained, and then the alignment is completed through the attention mechanism: ; ; ; ; in, is the weight matrix, Based on and The sequence mask matrix is ​​calculated based on the actual sequence length, thereby avoiding the influence of semantically irrelevant characters such as [CLS], [SEP], and [PAD] in text and visual data. Represents the query vector obtained by linear transformation of text features, Represents the key vector obtained by linear transformation of video features, Represents the value vector obtained by linear transformation of video features.

[0027] The sequence mask matrix of this embodiment To generate , first multiply the mask of the text modality and the mask of the visual modality by matrix multiplication to generate a mask matrix with the same shape as the attention matrix. This mask matrix reflects the attention relationship between elements of different modalities. Then, the positions of the special characters [CLS], [SEP], and [PAD] in the mask are all set to a very large negative number to ensure that these characters do not participate in the attention calculation, and the other characters that need to participate in the attention calculation are set to 0. The mask of each modality is a binary matrix composed of 1 and 0, with a dimension of , 1 means the original modality has meaning at this location, and 0 means the original modality sequence is a supplementary meaningless character at this location.

[0028] Finally, the processed mask matrix is ​​added to the original attention matrix through The function converts it into a probability distribution and normalizes the attention weight. The position of the extremely negative number in the attention matrix is The function will produce an attention weight close to 0, which successfully suppresses irrelevant attention.

[0029] Similarly, audio features The same attention method can also be used with Complete word-level alignment and get .

[0030] The word-level aligned features of each modality are fed into a single-layer unidirectional Gated Recurrent Unit (GRU) to extract the temporal features of each modality: ; .

[0031] Step S3: splice the temporal features of each modality in the time dimension, and then input the spliced ​​multimodal sequence into the multimodal composite module to obtain composite features.

[0032] The multimodal composite module (MCT) proposed in this embodiment is used to efficiently complete the interactive fusion of single modality, dual modality, and trimodality. First, the temporal features of the three modalities are spliced ​​in the time dimension: Then, in order to effectively complete the interactive fusion of single modality, dual modality and trimodality, the spliced ​​multimodal sequence Feed into the proposed multimodal composite module MCT.

[0033] The multimodal composite module of this embodiment includes N Multimodal composite layers, such as Figure 3 As shown, each multimodal composite layer is connected in sequence. i A multimodal composite layer performs the following operations: After the input features are normalized in the first layer, the first features are obtained; The first feature is input into the multimodal hybrid attention module to obtain the second feature; After the first feature is added to the second feature, the third feature is obtained after the second layer normalization; The third feature is added to the third feature after passing through the feedforward network to obtain the output feature of the multimodal composite layer.

[0034] It is expressed by the following formula: ; ; ; in, LN is layer normalization (LN), FFN is a feed-forward network (Feed-ForwardNetwork), The features output by the last multimodal composite layer are the final composite features. MCA Represents a multimodal mixed attention module MCA The action performed.

[0035] Among them, the main component of the multimodal composite layer is the multimodal composite attention module (MCA). MCA In the module, perform the following operations: Step 3.1: For the input features, calculate the self-attention matrix.

[0036] That is, for input , first complete the calculation of the self-attention matrix: ; in, 、 is the weight parameter. is the query vector obtained by linear transformation of the input features, It is the key vector obtained by linear transformation of the input features.

[0037] Step 3.2: Add the self-attention matrix to each unimodal and bimodal mask matrix to generate each unimodal attention matrix and bimodal attention matrix.

[0038] In this embodiment, the self-attention matrix They are added to 6 different mask matrices respectively to form unimodal and bimodal interactions in various situations, and the unimodal attention matrices and bimodal attention matrices are obtained.

[0039] Text unimodal mask matrix , where the position of the text modality self-attention is 0, and the rest are extremely negative numbers; the audio unimodal mask matrix , where the position of the audio modality self-attention is 0, and the rest are extremely negative numbers; the visual unimodal mask matrix , the position of the visual modality self-attention is 0, and the rest are extremely large negative numbers.

[0040] Text-audio bimodal mask matrix , the mutual attention positions between the text modality and the audio modality, the self-attention positions of the text modality, and the self-attention positions of the audio modality are 0, and the rest are extremely large negative numbers; Text-Vision Bimodal Mask Matrix , the positions where the text modality and the visual modality pay attention to each other, the positions where the text modality pays attention to itself, and the positions where the visual modality pays attention to itself are 0, and the rest are extremely negative numbers; Audio-visual bimodal mask matrix , the positions where the audio modality and the visual modality pay attention to each other, the positions where the audio modality pays attention to itself, and the positions where the visual modality pays attention to itself are 0, and the rest are extremely large negative numbers.

[0041] To generate various mask matrices, first generate an all-0 mask matrix with the same shape as the self-attention matrix, and then according to the actual sequence length, we can get the position that needs to be set to a very large negative number. This is a relatively mature technology in this field and will not be repeated here.

[0042] The final unimodal attention matrices are expressed as , where the subscript uni Indicates any single mode, subscript l 、 a 、 vRepresent text, audio and visual modalities respectively. The bimodal attention matrix is ​​expressed as , where the subscript bi represents any bimodal state, e.g. The following shows the dual modality of text and audio.

[0043] The self-attention matrix They are added to 6 different mask matrices respectively, and the formula is as follows: ; ; ; ; ; .

[0044] Step 3.3: Make the self-attention matrix, each unimodal attention matrix and bimodal attention matrix softmax After the operation, it is multiplied by the value vector corresponding to the input feature to obtain the corresponding attention representation.

[0045] Specifically, Do softmax After operation, Multiplying them together completes the attention calculation, resulting in a total of 7 multimodal attention representations: ; in, It is the value vector of each input feature after linear transformation.

[0046] Step 3.4: Map the attention representation through the linear layer, then extract the effective parts of the mapped features separately, and splice them in the feature dimension to obtain the spliced ​​features of the three modalities of text, audio, and vision.

[0047] because A total of 7 types of matrices are derived from the same attention matrix Therefore, in order to enable the different modes to have essential interactions and avoid invalid interactions, Seven different modal interaction-specific linear layers are fed into the layers for mapping: ; in 、 、 , respectively represented as 7 multimodal representations The customized linear layer, 、 、 They represent seven multimodal representations after being mapped by specific linear layers.

[0048] Next, The effective parts of the multimodal sequence are taken out separately and concatenated in the feature dimension: ; ; ; in, express The modal ( ) part, express The modal ( ) part, express The modal ( ) part, express The modal ( ) part, express The modal part (that is, itself), express The modal part (that is, itself), express The modal part (that is, itself).

[0049] Step 3.5: After concatenating the three modal features, the features are reduced in dimension through a linear layer, then concatenated in the time dimension, and then passed through a linear layer to obtain the final output.

[0050] This step will Send them to the corresponding linear layer for dimensionality reduction: ; Finally, The final MCA output is obtained after concatenation in the time dimension and passing through a linear layer: ; ; Finally, after After layer MCT, you can get .

[0051] Step S4: In the joint representation module, each modal part of the composite feature is extracted separately, and the attention mechanism operation is performed to obtain the discourse-level representation of each modality. The discourse-level representation of each modality is then concatenated and passed through a multi-layer perceptron to obtain a multimodal joint representation.

[0052] For the output of MCT , take out the part of each mode and get , and then through the most basic attention mechanism Processing is performed to complete the aggregation in the time dimension and obtain the discourse-level representation of each modality : ; in It is the most basic attention mechanism (Naive Attention, NA).

[0053] Then, the discourse-level representations of each modality are concatenated and passed through a multi-layer perceptron to obtain a multimodal joint representation. This step is shown in the following formula: ; ; in, Indicates splicing.

[0054] Step S5: Input the multimodal joint representation into the classification module to obtain the recognition result.

[0055] Finally, the multimodal joint representation is input into the classification module, which can use a commonly used fully connected layer to obtain the final recognition result.

[0056] Example 2: Based on Example 1, this example alleviates label deviation.

[0057] For label deviation, this embodiment chooses to mitigate it based on backdoor adjustment. Specifically, the traditional prediction pass Operators intervene, where is the multimodal fusion function, adjusted to In simple terms, it is to traverse the confounding factors All situations involved are weighted and summed with the probabilities of their corresponding events to obtain a more robust prediction.

[0058] In implementation, this embodiment proposes improvements to the classification module, such as Figure 4 As shown, the classification module performs the following operations: Step 6.1: Use a multi-layer perceptron to encode the prior features of each modality to obtain the encoded features of each modality.

[0059] In this embodiment, the prior features of each modality are feature means of each modality feature extracted by the feature extraction module.

[0060] It should be noted that during training, the total number of training samples is By calculating the feature mean of each mode in each class, the prior features of each mode can be obtained. 、 and ,in In testing or actual applications, when performing emotion recognition on users, the mean of all input samples is directly used as the prior features of each modality, and the category is no longer considered. Therefore, the prior features of each modality of each class are the same. In addition, the prior probability of each class is They are also 1 / cls (assuming uniform distribution).

[0061] In order to make the low-level modal prior features Able to be jointly represented with high-level multimodal To interact, this step uses the modality-shared chopped perceptron MLP to encode it and obtain the encoding features of each modality: ; in Both are composed of three linear layers and two layers of ReLU activation functions.

[0062] Step 6.2: Concatenate the encoding features of each modality with the multimodal joint representation and input them into the fully connected layer to obtain the preliminary classification results.

[0063] In this step, each modal encoding feature Joint representation with multimodal Perform splicing to obtain causal representation: ; Then, the causal representation Input to the fully connected layer for classification and obtain the category The preliminary classification results under ,Right now item.

[0064] Step 6.3: Multiply the preliminary classification result by the prior probability and sum them to obtain the recognition result after label bias correction.

[0065] Compare the preliminary classification results with the pre-calculated prior probability of each class label distribution Multiply and sum to get a robust classification output : .

[0066] Example 3: This example, based on Example 1 or 2, alleviates language bias.

[0067] For language bias, we choose to build fine-grained counterfactual text by calculating the coefficient of variation of words in the dataset, and subtract the recognition result of the counterfactual text from the text recognition result output by the classification module to obtain the final recognition result. : ; in Represents the text recognition result output by the classification module, represents the recognition result of the counterfactual text output by the classification module, . is a hyperparameter that controls the extent to which counterfactual predictions are subtracted.

[0068] In a specific embodiment, the construction of fine-grained counterfactual text is specifically described. process.

[0069] First, calculate the number of words in the dataset. The number of occurrences in each class, get the set : ; Then, by collecting To calculate the word The coefficient of variation is: ; in, is the standard deviation, is the mean.

[0070] Next, define When The inter-class distribution difference is large, so in the counterfactual text In order to avoid some words that appear less frequently in the dataset but are classified as counterfactual words due to their uneven distribution between classes, only the top 100 words are considered for counterfactual text construction in this application. Or a word with fewer words, replace it with the [MASK] character.

[0071] In summary, after the above series of operations, we can get The counterfactual text is then passed through the same feature extraction module to obtain the counterfactual text features. .

[0072] Then, the counterfactual text features The network model obtains the recognition result of the counterfactual text output by the classification module, and subtracts the recognition result after the counterfactual text is replaced from the recognition result output by the classification module to obtain the final recognition result. , thereby alleviating language bias.

[0073] Example 4: In order to improve the robustness of the model in modal missing scenarios without increasing the amount of additional training and keeping the model lightweight, this application uses hierarchical self-distillation to perform distillation from three dimensions: low-level features, attention maps, and high-level joint representations from the bottom up to improve the robustness of the model in modal missing scenarios.

[0074] When training the network model, the training samples include incomplete data samples and complete data samples, and the corresponding losses are calculated in three dimensions: low-level features, attention maps, and high-level joint representations, and participate in the update of model parameters.

[0075] In the case of low-level feature dimensions and incomplete data sample input, the output of MCT Reconstruct and obtain the reconstructed features : ; ; ; Then use the regression loss function SmoothL1 loss to Output low-level features of the feature extraction module Perform alignment and calculate the corresponding reconstruction loss as follows: ; In the attention graph dimension, the self-attention matrix of the last layer of MCT under the complete data sample and incomplete data sample input is taken out respectively to obtain and . And use Kullback-Leibler (KL) divergence for alignment, and calculate the corresponding KL divergence loss as follows: ; In the high-level joint representation dimension, the multimodal joint representations under the complete data sample and the incomplete data sample input are taken out respectively to obtain and . And use cosine similarity for alignment, and calculate the corresponding cosine similarity loss as follows: ； in, Calculates cosine similarity.

[0076] Then, in order to fully utilize the information of complete and incomplete modalities during the training process, this application designs a two-stage optimization method.

[0077] In the first stage, the complete data sample is input, the cross entropy loss corresponding to the classification module output is calculated, and then back propagation is performed to update the model parameters. The cross entropy loss is expressed as follows: ; in, is the cross entropy loss function, represents the calculated loss, To identify demerits, For label.

[0078] In the second stage, incomplete data samples are input and the cross entropy loss corresponding to the classification module output is calculated. The cross entropy loss is expressed as follows: ; in, is the cross entropy loss function, represents the calculated loss, To identify demerits, For label.

[0079] Then the three losses mentioned above are combined with Perform joint optimization to obtain the overall optimization objective for incomplete modal input: ; in, 、 and They are hyperparameters used to control the degree of distillation at the three levels, and finally back propagation is performed to update the model parameters.

[0080] Example 5: This example proposes a multimodal emotion recognition device for modality-missing and out-of-distribution data, including a processor and a memory storing a plurality of computer instructions, wherein the computer instructions implement the steps of the above method when executed by the processor.

[0081] Regarding the specific limitations of the multimodal emotion recognition device for modal missing and out-of-distribution data, please refer to the limitations of the multimodal emotion recognition method for modal missing and out-of-distribution data above, which will not be repeated here. The above-mentioned multimodal emotion recognition device for modal missing and out-of-distribution data can be implemented in whole or in part by software, hardware, and a combination thereof. It can be embedded in or independent of the processor in the computer device in hardware form, or it can be stored in the memory of the computer device in software form, so that the processor can call and execute the above corresponding operations.

[0082] The memory and the processor are electrically connected, directly or indirectly, to enable data transmission or interaction. For example, these elements may be electrically connected to each other via one or more communication buses or signal lines. The memory stores a computer program executable on the processor. The processor executes the computer program stored in the memory to implement the multimodal emotion recognition method for modality-missing and out-of-distribution data in the embodiments of the present invention.

[0083] The memory may be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory is used to store a program, and the processor executes the program after receiving an execution instruction.

[0084] The processor may be an integrated circuit chip with data processing capabilities. The processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor.

[0085] This application also experimentally validates the technical solution of this application. The experiments were conducted on the CMU-MOSI and CMU-MOSEI datasets, comparing them with current mainstream multimodal emotion recognition algorithms in the presence of modal missing and out-of-distribution data. Performance was measured using the binary classification accuracy (Acc2), the binary classification F1 score (F1), and the seven-category accuracy (Acc7), with higher performance being preferred.

[0086] The experimental results are shown in Tables 1 to 6: Table 1 Table 1 shows the performance comparison experimental data in the case of modal missing on the CMU-MOSI dataset.

[0087] Table 2 Table 2 shows the performance comparison experimental data in the case of mode missing on the CMU-MOSEI dataset.

[0088] Table 3 Table 3 shows the performance comparison experimental data of the out-of-distribution data on the CMU-MOSI dataset.

[0089] Table 4 Table 4 shows the performance comparison experimental data of the out-of-distribution data on the CMU-MOSEI dataset.

[0090] Table 5 Table 5 shows the performance comparison experimental data in the case of modal missing and out-of-distribution data on the CMU-MOSI dataset.

[0091] Table 6 Table 6 shows the performance comparison experimental data in the case of modal missing and out-of-distribution data on the CMU-MOSEI dataset.

[0092] Through the comparison of the experimental data in the above six tables, it can be seen that the multimodal emotion recognition performance of the present application method in various situations is mostly better than other multimodal emotion recognition algorithms from 2022 to 2025. Therefore, the present application method has certain advantages over other algorithms.

[0093] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A multimodal emotion recognition method for modality-missing and out-of-distribution data, which performs emotion recognition by constructing and training a multimodal emotion recognition network model, characterized in that: The multimodal emotion recognition network model includes a feature extraction module, a word-level self-aligned attention module, a multimodal composite module, a joint representation module, and a classification module. The multimodal emotion recognition method for modality-missing and out-of-distribution data includes: Obtain user text, audio, and visual data, and extract features of each modality through the feature extraction module; A word-level self-aligned attention module is used to adaptively align the features of each modality at the word level, and then the temporal features corresponding to the aligned modal features are extracted; The temporal features of each modality are spliced ​​in the time dimension, and then the spliced ​​multimodal sequence is input into the multimodal composite module to obtain the composite features; In the joint representation module, each modality of the composite feature is extracted separately and an attention mechanism is applied to obtain a discourse-level representation of each modality. The discourse-level representations of each modality are then concatenated and passed through a multi-layer perceptron to obtain a multimodal joint representation. Input the multimodal joint representation into the classification module to obtain the recognition result; Wherein, the multimodal composite module includes N Multimodal composite layers, each multimodal composite layer is connected in sequence. i A multimodal composite layer performs the following operations: After the input features are normalized in the first layer, the first features are obtained; The first feature is input into the multimodal hybrid attention module to obtain the second feature; After the first feature is added to the second feature, the third feature is obtained after the second layer normalization; The third feature is added to the third feature after passing through the feedforward network to obtain the output feature of the multimodal composite layer; The multimodal hybrid attention module performs the following operations: For the input features, calculate the self-attention matrix; Add the self-attention matrix to each unimodal and bimodal mask matrix to generate each unimodal attention matrix and bimodal attention matrix; The self-attention matrix, each unimodal attention matrix and bimodal attention matrix are respectively softmax After the operation, it is multiplied by the value vector corresponding to the input feature to obtain the corresponding attention representation; The attention representation is mapped through a linear layer, and then the effective parts of the mapped features are extracted separately and spliced ​​in the feature dimension to obtain the spliced ​​features of the three modalities of text, audio and vision; After the three modal features are spliced ​​together, they are reduced in dimension through a linear layer, then spliced ​​together in the time dimension, and then passed through a linear layer to obtain the final output.

2. The multimodal emotion recognition method for modality-missing and out-of-distribution data according to claim 1, characterized in that: The classification module performs the following operations: A multi-layer perceptron is used to encode the prior features of each modality to obtain the coded features of each modality; The encoding features of each modality are concatenated with the multimodal joint representation and then input into the fully connected layer to obtain the preliminary classification results; The preliminary classification result is multiplied by the prior probability and summed to obtain the recognition result after label deviation correction.

3. The multimodal emotion recognition method for modality-missing and out-of-distribution data according to claim 2, characterized in that: The multimodal emotion recognition method for modality-missing and out-of-distribution data includes: The recognition result of the counterfactual text is subtracted from the text recognition result output by the classification module to obtain the final recognition result.

4. The multimodal emotion recognition method for modality-missing and out-of-distribution data according to claim 1, characterized in that: When training the multimodal emotion recognition network model, the input training samples include incomplete data samples and complete data samples, and the following losses are calculated during training: In the case of incomplete data sample input, the composite features are reconstructed to obtain the corresponding reconstructed features, and then the regression loss function is used to calculate the reconstruction loss between the reconstructed features and the features of each modality extracted by the feature extraction module; In the case of complete data samples and incomplete data samples, the self-attention matrix output by the last multimodal composite layer is obtained respectively, and then the KL divergence loss of the self-attention matrix is ​​calculated; The cosine similarity loss of the multimodal joint representation is calculated when both complete and incomplete data samples are input.

5. The multimodal emotion recognition method for modality-missing and out-of-distribution data according to claim 4, characterized in that: When training the multimodal emotion recognition network model, there are two stages of calculating losses and updating model parameters, where: In the first stage, the complete data sample is input, the cross entropy loss corresponding to the recognition result output by the classification module is calculated, back propagation is performed, and the model parameters are updated; In the second stage, incomplete data samples are input, and the cross entropy loss corresponding to the recognition result output by the classification module is calculated. The cross entropy loss is combined with the reconstruction loss, KL divergence loss, and cosine similarity loss to form a joint loss, and back propagation is performed to update the model parameters.

6. A multimodal emotion recognition device for modality-missing and out-of-distribution data, comprising a processor and a memory storing a plurality of computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images