Multi-modal emotion recognition method and device

Abstract

The invention provides a multi-modal emotion recognition method and device, and relates to the technical field of artificial intelligence, and the method comprises the steps: carrying out the fusion of a first text modal feature and a first image modal feature through a cross attention mechanism, and generating an attention-fused image feature, fusing the first text modal feature and the first voice modal feature through a cross attention mechanism to generate an attention-fused voice feature; sending the voice features fused with the attention, the image features fused with the attention and the first text modal features into a multi-layer Transform encoder for learning to generate second voice modal features, second image modal features and second text modal features; and emotion recognition is carried out based on the second voice modal feature, the second image modal feature and the second text modal feature, so that accurate multi-modal emotion recognition is realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal emotion recognition method and device. Background Art

[0002] In recent years, affective computing has become a hot topic due to its ability to enhance multiple fields such as mental health monitoring, human-computer interaction, and personalized advertising. The advancement of affective computing is widely supported by sub-fields such as sentiment analysis and emotion recognition. In addition, deep learning (DL) techniques have made significant progress in the field of emotion recognition, giving rise to multimodal emotion recognition (MER) systems that can effectively process data from various sources such as audio, video, and text. However, despite the great progress, MER systems still face some challenges.

[0003] However, due to the complex interactions between different modalities in multimodal emotion recognition and the diversity of emotional information, existing methods have the following three problems, which make them unable to meet the existing emotion recognition needs based on pre-trained models:

[0004] It is difficult to improve the efficiency of acoustic features in multimodal emotion recognition, and their contribution is relatively small compared to vision and text: In emotion recognition tasks, acoustic information (such as speech features) often does not contribute as much to emotion recognition as visual information (such as facial expressions) and text information (such as language content). The extraction and processing of acoustic features are complex, and their effectiveness is greatly affected by external factors such as noise and recording equipment, making improving acoustic efficiency a major challenge.

[0005] It is difficult for the model to direct attention to the emotion-related segments in acoustic and visual information: In multimodal emotion recognition, how to make the model focus on the key emotion-related segments in acoustic and visual information is a difficult problem. Although the pre-trained model performs well in text emotion recognition, in multimodal tasks, the model often has difficulty coordinating the features of each modality, resulting in distraction and failure to fully utilize acoustic and visual information for emotion recognition.

[0006] Therefore, finding an accurate multimodal emotion recognition method for computer-assisted judgment of human emotions has always been a hot topic and difficulty in the field of multimodal emotions. Summary of the invention

[0007] In order to solve the technical problem that the multimodal emotion recognition in the prior art is not accurate enough, the embodiment of the present invention provides a multimodal emotion recognition method and device. The technical solution is as follows:

[0008] On the one hand, a multimodal emotion recognition method is provided, the method comprising:

[0009] Collecting text data of current emotional expression, and constructing a first text modality feature based on the text data using a pre-trained model;

[0010] Collecting speech data of current emotional expression, and constructing a first speech modality feature based on the speech data using a convolutional neural network, a bidirectional long short-term memory neural network, and a pre-trained model;

[0011] Collect image data of current emotional expression, and construct a first image modality feature based on the image data using a preset convolutional network;

[0012] The first text modality feature is fused with the first image modality feature through a cross attention mechanism to generate an image feature of fused attention, and the first text modality feature is fused with the first speech modality feature through a cross attention mechanism to generate a speech feature of fused attention;

[0013] The speech features with fused attention are sent to a multi-layer Transformer encoder for learning to generate a second speech modality feature, the image features with fused attention are sent to a multi-layer Transformer encoder for learning to generate a second image modality feature, the first text modality feature is sent to a multi-layer Transformer encoder for learning to generate a second text modality feature, and emotion recognition is performed based on the second speech modality feature, the second image modality feature and the second text modality feature.

[0014] Optionally, constructing the first text modality feature using the pre-trained model Bert-base-multilingual-uncased-sentiment includes:

[0015] First, use BERT's tokenizer to split the text into words or subwords;

[0016] Add tags to the split text, convert the text into a vocabulary index, and generate an input tensor, wherein the input tensor includes a vocabulary index tensor, a position encoding tensor, and a segment encoding tensor;

[0017] The input tensor is fed into the pre-trained Bert-base-multilingual-uncased-sentiment model, passed through a multi-layer Transformer encoder for contextual representation learning, and the labeled output vector is extracted as the first text modality feature.

[0018] Optionally, constructing the first speech modality feature based on the speech data using a convolutional neural network, a bidirectional long short-term memory neural network and a pre-trained model includes:

[0019] Read the speech data through three channels of spectrogram, MFCC and original audio data;

[0020] Use the ResNet network to extract spectrogram features from the spectrogram, use the BiLSTM network to extract MFCC features from the MFCC data, and use the wav2vec 2.0 network to extract raw audio features from the raw audio data;

[0021] Merging the spectrogram features with the MFCC features to generate merged features, and refining the merged features through a linear layer to generate weights of wav2vec 2.0 features in different frames, and weighting the original audio features according to the weights to generate weighted audio features;

[0022] The spectrogram features, the MFCC features and the weighted audio features are connected to construct a comprehensive acoustic feature.

[0023] Optionally, constructing a first image modality feature based on the image data using a preset convolutional network includes:

[0024] extracting a frame image from the image data;

[0025] Use MTCNN to identify facial information in frame images and locate facial key points;

[0026] The frame image after locating the facial key points is sent to a preset convolutional network to extract the first image modality features. The preset convolutional network includes multiple convolutional layers, a RepVGG Block module, a batch normalization layer, an activation function layer and a maximum pooling layer.

[0027] Optionally, the fusing the first text modality feature with the first image modality feature through a cross attention mechanism to generate an image feature with fused attention, and fusing the first text modality feature with the first speech modality feature through a cross attention mechanism to generate a speech feature with fused attention, comprises:

[0028] Using the first image modality feature as a first query vector feature map, and the first text modality feature as a queried feature map;

[0029] Perform cross attention calculation on the first query vector feature map and the queried feature map, select the first p parts of the probability in each attention score map by kernel sampling method and perform broadcast point-by-point multiplication with the original feature map to obtain image features of fused attention;

[0030] The first speech modality feature is used as a second query vector feature graph, and the first text modality feature is used as a queried feature graph;

[0031] Perform cross attention calculation on the second query vector feature map and the queried feature map, select the first p parts of the probability in each attention score map and perform broadcast point-by-point multiplication with the original feature map by the kernel sampling method to obtain speech features of fused attention;

[0032] The P is a positive integer.

[0033] Optionally, the multimodal emotion recognition method further includes:

[0034] The machine learning model is trained through gradient calculation and batch normalization to implement the multimodal emotion recognition method using the machine learning model.

[0035] Optionally, the loss function of the machine learning model is a binary cross entropy loss function.

[0036] On the other hand, a multimodal emotion recognition device is provided, and the multimodal emotion recognition device is used to implement the multimodal emotion recognition method provided by an embodiment of the present invention, and the device includes:

[0037] A first acquisition module, used for acquiring text data of current emotional expression, and constructing a first text modality feature based on the text data using a pre-trained model;

[0038] A second acquisition module is used to collect voice data of current emotional expression, and construct a first voice modality feature based on the voice data using a convolutional neural network, a bidirectional long short-term memory neural network and a pre-trained model;

[0039] A third acquisition module is used to acquire image data of current emotion expression, and construct a first image modality feature based on the image data using a preset convolutional network;

[0040] A fusion module is configured to fuse the first text modality feature with the first image modality feature through a cross attention mechanism to generate an image feature of fused attention, and to fuse the first text modality feature with the first speech modality feature through a cross attention mechanism to generate a speech feature of fused attention;

[0041] An emotion recognition module is used to send the speech features of the fused attention into a multi-layer Transformer encoder for learning to generate a second speech modality feature, send the image features of the fused attention into a multi-layer Transformer encoder for learning to generate a second image modality feature, send the first text modality feature into a multi-layer Transformer encoder for learning to generate a second text modality feature, and perform emotion recognition based on the second speech modality feature, the second image modality feature and the second text modality feature.

[0042] On the other hand, a multimodal emotion recognition device is provided, the multimodal emotion recognition device comprising:

[0043] processor;

[0044] A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the method provided by the embodiment of the present invention.

[0045] On the other hand, a computer-readable storage medium is provided, in which a program code is stored. The program code can be called by a processor to execute the method provided by an embodiment of the present invention.

[0046] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0047] The embodiment of the present invention fuses the first text modal feature with the first image modal feature through a cross-attention mechanism to generate an image feature of fused attention, and fuses the first text modal feature with the first speech modal feature through a cross-attention mechanism to generate a speech feature of fused attention; the speech feature of fused attention is sent to a multi-layer Transformer encoder for learning to generate a second speech modal feature, the image feature of fused attention is sent to a multi-layer Transformer encoder for learning to generate a second image modal feature, the first text modal feature is sent to a multi-layer Transformer encoder for learning to generate a second text modal feature, and emotion recognition is performed based on the second speech modal feature, the second image modal feature and the second text modal feature, thereby achieving more accurate multimodal emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 is a flow chart of a multimodal emotion recognition method provided by an embodiment of the present invention;

[0050] Figure 2 is a flow chart of obtaining a first speech modality feature provided by an embodiment of the present invention;

[0051] Figure 3 is a flow chart of obtaining a first image modality feature provided by an embodiment of the present invention;

[0052] Figure 4It is a schematic diagram of the results of the comparison between the embodiment of the present invention and the SOTA model on the IEMOCAP dataset;

[0053] Figure 5 It is a schematic diagram of the comparison results between the embodiment of the present invention and the SOTA model on the CMU-MOSEI dataset;

[0054] Figure 6 is a block diagram of a multimodal emotion recognition device provided by an embodiment of the present invention;

[0055] Figure 7 It is a structural schematic diagram of a multimodal emotion recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0057] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0058] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0059] In the embodiments of the present invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are consistent.

[0060] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0061] In order to solve the technical problem that the multimodal emotion recognition in the prior art is not accurate enough, the embodiment of the present invention provides a multimodal emotion recognition method and device. The technical solution is as follows:

[0062] On the one hand, a multimodal emotion recognition method is provided, such as Figure 1 As shown, the method includes:

[0063] S1, collecting text data of current emotional expression, and constructing a first text modality feature based on the text data using a pre-trained model;

[0064] S2, collecting voice data of current emotional expression, and constructing a first voice modality feature based on the voice data using a convolutional neural network, a bidirectional long short-term memory neural network and a pre-training model;

[0065] S3, collecting image data of current emotion expression, and constructing a first image modality feature based on the image data using a preset convolutional network;

[0066] S4, fusing the first text modality feature with the first image modality feature through a cross attention mechanism to generate an image feature of fused attention, and fusing the first text modality feature with the first speech modality feature through a cross attention mechanism to generate a speech feature of fused attention;

[0067] S5. Send the speech features with fused attention into a multi-layer Transformer encoder for learning to generate a second speech modality feature, send the image features with fused attention into a multi-layer Transformer encoder for learning to generate a second image modality feature, send the first text modality feature into a multi-layer Transformer encoder for learning to generate a second text modality feature, and perform emotion recognition based on the second speech modality feature, the second image modality feature and the second text modality feature.

[0068] In the embodiment of the present invention, the working process of S1 is as follows:

[0069] S11, load the pre-trained model:

[0070] Use the AutoModel.from_pretrained method in the Transformers library to load the bert-base-multilingual-uncased-sentiment pre-trained model.

[0071] S12, text data processing:

[0072] The text data is the text data expressing the current emotion.

[0073] Next, some text data text is input into the bert-base-multilingual-uncased-sentiment pre-trained model loaded in S11. S12 is completed by calling the model instance and passing in the text data. The bert-base-multilingual-uncased-sentiment pre-trained model processes the text data and generates a series of outputs.

[0074] S13, feature extraction:

[0075] In the output of the bert-base-multilingual-uncased-sentiment pre-trained model, there is a key component called last_hidden_state. This is the hidden state of the last layer of the model, represented as a 3D tensor. For multiple text data (i.e., in a batch), the shape of this tensor is: batch size, sequence length, feature dimension.

[0076] The key part of each input sample is the first element of the sequence (usually corresponding to a special [CLS] token). The output of this [CLS] token is considered to be a summary representation of the entire input sequence and can be used for subsequent tasks such as classification.

[0077] To obtain the representation of these [CLS] tags, extract the first column from each sequence of last_hidden_state. This results in a new matrix with the shape of batch size, feature dimension. Each row in the matrix is ​​a [CLS] feature vector corresponding to an input sample.

[0078] S14, return result:

[0079] Finally, these [CLS] feature vectors are returned as the input of the text modality features of the subsequent multimodal sentiment recognition task, that is, the first text modality features.

[0080] In view of the problems that it is difficult to improve the efficiency of acoustic features in multimodal emotion recognition and their small contribution compared to vision and text, an embodiment of the present invention provides a method for extracting speech features using a multi-stage acoustic encoder, which can improve the contribution of the speech modality in multimodal emotion recognition.

[0081] In this embodiment, Figure 2 As shown, the workflow of the speech feature extraction module is as follows:

[0082] S21, reading the original audio file;

[0083] The original audio file is the voice data expressing the current emotion.

[0084] First, use Python's librosa library to read the original audio file and retain the original sampling rate. Apply pre-emphasis filtering to the read audio signal as a high-pass filter to enhance the high-frequency components, and use the filtered content as the original audio feature.

[0085] S22, obtain the spectrum features:

[0086] First, a set of spectrogram processing parameters are defined, including window type (Hamming window), window length (40 milliseconds), step length (10 milliseconds), Fourier transform length (800 points), frequency resolution (200 points), number of Mel frequency filters (128), audio segment size (300 points), and whether noise is mixed (Boolean flag). Subsequently, the window length and step length are converted from milliseconds to the number of sampling points. Next, the short-time Fourier transform (STFT) of the audio signal is determined to obtain the spectrogram. The spectrogram is then converted to a logarithmic magnitude spectrum, expressed in decibels. In order to extract the desired frequency components, only the first 200 frequency points are retained. The ResNet-50 network is then used to extract features from the spectrogram. After applying dropout and flattening the vector, it is passed to the linear layer and the ReLU activation function is used to enhance the feature representation.

[0087] S23, obtain MFCC features:

[0088] First, a set of MFCC processing parameters are defined, including the number of MFCCs (40), sampling rate, overlap between windows (10 sampling points), and HTK (Hidden Markov Model Toolkit) style. By executing the librosa.feature.mfcc function, the MFCC feature matrix of the audio signal is obtained. For the convenience of subsequent processing, the matrix is ​​transposed to make it have a shape of (T,n_mfcc), where T represents the number of time frames and n_mfcc represents the number of MFCC coefficients extracted. Then a BiLSTM network is used to find the temporal dynamics in the MFCC features. After applying dropout and flattening the vector, we pass it to a linear layer and use the ReLU activation function to enhance feature extraction.

[0089] S24, obtain Wav2vec2.0 audio features:

[0090] First, the pre-trained Wav2Vec 2.0 processor is loaded using the Wav2Vec2Processor class, which contains the configuration and feature extraction methods of the pre-trained model. The loaded processor is used to process the audio data. The raw audio data is provided in the form of a NumPy array with a sampling rate of 16000Hz. The processing method of wav_processor is called to convert the audio data into Wav2vec2.0 audio features.

[0091] S25, obtain Wav2vec2.0 weighted features:

[0092] First, weight values ​​are obtained by concatenating MFCC and spectrogram features, and then they are activated using a rectified linear unit (ReLU). The output of the wav2vec 2.0 model is refined using the generated weights, and weighted wav2vec 2.0 features are produced.

[0093] S26, fusion of speech features from different channels:

[0094] The spectrogram features, MFCC features, and weighted wav2vec 2.0 features are connected as the comprehensive acoustic features for emotion recognition, which are the first speech modality features.

[0095] The specific process of speech feature extraction algorithm is as follows:

[0096] Use default parameters wimdow=hamming, wimlength=40, hop_length=10, ndft=800, nfreq=200, nmel=128, n_mfcc=40

[0097] Input: The original audio files in the dataset.

[0098] Output: Speech features for multimodal emotion recognition.

[0099] Step 201: The original audio is loaded via the 1ibrosa library.

[0100] Step 202: Extract the original audio spectrum using preset parameters as S spec , and extract the feature S′ from the spectrogram by using the ResNet-50 network spec , which is the spectrogram feature. Among them, x represents the original audio feature after pre-emphasis, sr represents the sampling rate of the audio signal, and params represents the window type (Hamming window), window length (40 milliseconds), jump length (10 milliseconds), Fourier transform length (800 points), and frequency resolution (200 points).

[0101] S spec =extract_logspec(x,sr,params).

[0102] S′ spec =ReLU(ResNet50(S spec )).

[0103] Step 203: Extract the MFCC of the original audio using preset parameters as S mfcc , and extract features S′ from MFCC by using a BiLSTM network mfcc, which is the MFCC feature. Among them, x represents the original audio feature after pre-emphasis, sr represents the sampling rate of the audio signal, and n mfcc Indicates the number of MFCCs (40), hop_length indicates the overlap between windows (10 sampling points), and htk indicates the HTK (Hidden Markov Model Toolkit) style.

[0104] S mfcc =librosa.feature.mfcc(y=x,sr=sr,n mfcc =40,hoplength=hop_length,htk=True).

[0105] S′ mfcc =ReLU((BiLSTM(S mfcc )).

[0106] Step 204: Extract the Wav2vec2.0 features of the original audio using the pre-trained model Wav2vec2.0 as S' wav .

[0107] Step 205: Concatenate MFCC and spectrogram features to obtain weight values ​​to refine the output of the wav2vec 2.0 model.

[0108] S′ weight =ReLU(S′ mfcc ⊕S′ spec ), where S′ weight is the weight.

[0109] S′ w-wav =(S′ weight ·S′ wav ) T , where S′ w-wav is the weighted audio feature.

[0110] Step 206: Fuse speech features from different channels to obtain comprehensive acoustic features for emotion recognition.

[0111] y = ReLU(S′) mfcc ⊕S′ spec ⊕S′ w-wav ).

[0112] In order to solve the problem that the reasoning speed of high-definition video is slow and affects real-time applications, the embodiment of the present invention uses a reparameterized convolutional network (RepVGG) to extract image features, thereby increasing the efficiency of image feature extraction and improving the real-time application speed when performing emotion recognition on high-definition video.

[0113] In the embodiment of the present invention, Figure 3 As shown, the workflow of S3 is:

[0114] S31, read the original image file for processing and extract the frame image.

[0115] The original image file is the image data expressing the current emotion.

[0116] Calculate the number of images and determine the sampling step size based on the frame rate and image interval to generate a list of qualified image paths. For each image path, use the Image.open function in the PIL library to open the image file and convert the image data into a 32-bit floating-point NumPy array, which is the image array, in which the frame image is saved.

[0117] S32, extract the face position of the image:

[0118] The multi-task convolutional neural network (MTCNN) is used to detect and align faces in the image array. If a face is not detected in an image, the image is center-cropped, that is, the center offset is calculated and a center area of ​​a specified size (with a high probability of containing a face or part of a face) is cropped to enable the training process to proceed normally. The face position is the facial key point

[0119] S33, image feature extraction;

[0120] The image data containing facial information is extracted through a preset convolutional network and image features are obtained, which are the first image modality features. The preset convolutional network mainly includes multiple convolutional layers and RepVGG Block modules, combined with batch normalization layers, activation function (ReLU) layers, and maximum pooling (Max Pooling) layers. The introduction of the RepVGGBlock module is mainly aimed at simplifying complex convolution operations while maintaining the expressive power of the network. In the training phase, a complex network structure with multiple branches (such as 1x1 convolution and 3x3 convolution) is used to improve the training ability and performance of the model. In the inference phase, these complex branches are merged into an equivalent simple convolution layer to improve inference efficiency.

[0121] The image feature extraction algorithm process is as follows:

[0122] Input: image files in the dataset.

[0123] Output: Image features for multimodal emotion recognition.

[0124] Step 301: load the image through the PIL library, calculate the sampling step size based on the frame rate and image interval, and filter the appropriate image path list;

[0125] Step 302: For each selected image, use Image.open to open the image file and convert the image data into a 32-bit floating point NumPy array;

[0126] Step 303: Perform face detection and alignment through a multi-task convolutional neural network (MTCNN).

[0127] faces=MTCNN(imgs)

[0128] If a face is detected (face!=None), the detected face is used directly;

[0129] If no face is detected (face == None), a center cropping process is performed.

[0130] For images where no faces are detected, the image dimensions are first resized to meet the processing requirements:

[0131] img=img i permute(2,0,1)

[0132] Among them, img i Represents the i-th image in the original image list imgs.

[0133] Then perform center cropping. The logic of center cropping can be described as:

[0134] Croppedd=img[:,off:off+taget_size,off:off+target_size]

[0135] here,

[0136]

[0137] current_size is the size of the current image (assuming the image is square with equal height and width); target_size is the desired cropped image size; off is the calculated offset to ensure that the cropped area is in the center of the image.

[0138] Step 304: Use the designed neural network, including multiple convolutional layers, RepVGG Block modules, batch normalization, activation function (ReLU) and maximum pooling, to extract features from the image data containing the face;

[0139] Initialize the convolutional layer:

[0140] An initial convolution operation is applied to the input image to extract basic spatial features.

[0141] Batch Normalization and ReLU activation:

[0142] The features output by the convolutional layer are batch normalized, and then the nonlinearity is enhanced by the ReLU activation function.

[0143] ReLU(BN 64 (Conv2d out ))

[0144] Max Pooling:

[0145] Use max pooling to downsample features to reduce dimensionality and retain important information.

[0146] RepVGG Block sequence:

[0147] The features are further enhanced through a series of complex RepVGG Block modules to extract more advanced abstract features.

[0148] Final max pooling:

[0149] After a series of RepVGG Block modules, the final features are downsampled again using maximum pooling.

[0150] MaxPool2d(kernel_size=2, stride=2)(final_out)

[0151] After this series of sequences, the final_out contains high-level features that have been processed and abstracted in multiple layers, which is suitable for subsequent multimodal emotion recognition tasks. Through the design of complex network structure, the neural network can effectively extract deep and key facial features in the image, providing strong feature support for multimodal emotion recognition.

[0152] In multimodal emotion recognition, how to focus on the key emotion-related segments in acoustic and visual information is a difficult problem. Although the pre-trained model performs well in text emotion recognition, it is often difficult to coordinate the features of each modality in multimodal tasks, resulting in distraction and failure to fully utilize acoustic and visual information for emotion recognition. The example of the present invention uses a multimodal feature fusion module to fuse different modal features using a cross-attention mechanism, and uses an attention mechanism to dynamically adjust the weight of each modal information.

[0153] In the embodiment of the present invention, the working process of S4 is as follows:

[0154] Using the first image modality feature as a first query vector feature map, and the first text modality feature as a queried feature map;

[0155] Perform cross attention calculation on the first query vector feature map and the queried feature map, select the first p parts of the probability in each attention score map by kernel sampling method and perform broadcast point-by-point multiplication with the original feature map to obtain image features of fused attention;

[0156] The first speech modality feature is used as a second query vector feature graph, and the first text modality feature is used as a queried feature graph;

[0157] Perform cross attention calculation on the second query vector feature map and the queried feature map, select the first p parts of the probability in each attention score map and perform broadcast point-by-point multiplication with the original feature map by the kernel sampling method to obtain speech features of fused attention;

[0158] The P is a positive integer.

[0159] Cross-attention includes two inputs: (1) the query vector (q) belongs to the space (R d ). This vector feature map acts as a pointer and will actively search for relevant information in another given input. (2) A series of feature maps (M), where M∈R (C ×S×H×W) , C, S, H, and W represent the number of channels, sequence length, and the dimensions of height and width, respectively. Use q to perform spatial cross attention on the feature map for modality fusion. The query vector is the query vector feature map.

[0160] The multimodal feature fusion algorithm process is as follows:

[0161] Input: preprocessed speech features, image features, and text features.

[0162] Output: deep speech features, image features, and text features fused with attention.

[0163] Step 401: Query vector (q) transformation:

[0164] First, the query vector q is transformed through a linear transformation layer to obtain a new query vector q′. Here q belongs to the space R d , the purpose of this transformation is to adjust the representation of the query vector to make it more suitable for subsequent processing. The formula is as follows:

[0165] q′=Limear(q)

[0166] Step 402: Transformation of feature map (M): Then, a linear transformation is performed on the feature map M. Here, the feature map M is a four-dimensional tensor, that is, M∈R C×S×H×W , where C, S, H, and W represent the number of channels, sequence length, height, and width, respectively. Before transforming M, it will be rearranged. After the transformation, M′ is obtained. The formula is as follows:

[0167] M′=Linear(permute(M))

[0168] Step 403: Calculation: Next, the transformed q′ and M′ are combined by a specific operation (here represented by addition ⊕) and the hyperbolic tangent function (tanh) is applied to obtain M k This step is to fuse the information of the query vector and the feature map. The formula is as follows:

[0169] M k =tanh(M′⊕q′)

[0170] Step 404: Calculation: By calculating M k Apply a linear transformation and normalize using the softmax function to generate M i This will allow M k The element values ​​of are converted into a series of weights, which indicate the importance of each feature in information fusion. The formula is as follows:

[0171] M i =softmax(Linear(M k ))

[0172] Step 405: Nucleus Sampling: Next, i Perform nucleus sampling to obtain M n Kernel sampling is a commonly used sampling method in generative models to select the most likely choice from a probability distribution. The formula is as follows:

[0173] M n =NucleusSampling(M i )

[0174] Step 406: Application of attention mechanism: Finally, M n And the original feature map M performs specific operations (here M att Represents) combination to obtain the final attention weighted feature map M att This step essentially uses the calculated attention weights to select and emphasize the most important features. The formula is as follows:

[0175]

[0176] The speech features of fused attention and the image features of fused attention are fed into the multi-layer stacked Transformer encoders with the text modality features. Through the multi-layer processing of the encoder, the information of each modality is further fused and refined to generate a richer and more discriminative representation. This can better understand and integrate information from different sources, thereby making more accurate predictions or generation.

[0177] The speech features with fused attention are sent to a multi-layer Transformer encoder for learning to generate a second speech modality feature, the image features with fused attention are sent to a multi-layer Transformer encoder for learning to generate a second image modality feature, the first text modality feature is sent to a multi-layer Transformer encoder for learning to generate a second text modality feature, and emotion recognition is performed based on the second speech modality feature, the second image modality feature and the second text modality feature.

[0178] In an embodiment of the present invention, optionally, a machine learning model can be trained through gradient calculation and batch normalization to implement the multimodal emotion recognition method using the machine learning model.

[0179] Step 501, machine learning model setting:

[0180] First, set the machine learning model to training mode to enable gradient calculation and batch normalization. Set the MTCNN part of the machine learning model to evaluation mode.

[0181] Step 502, data loading:

[0182] Prepare to read the training data by getting the dataloader corresponding to the 'train' key from the dataloaders dictionary.

[0183] Step 503, iterate the data loader:

[0184] For each batch in the data loader, the data is obtained through iteration: including data unique identifier (utteranceId), image (imgs), text (text), spectrogram features (specgrams), MFCC features (mfccs), Wav2Vec features (wav2vecs), and labels (Y).

[0185] Step 504, data and model preparation:

[0186] Before performing gradient updates, clear the gradients of all parameters in the model.

[0187] Step 505, forward propagation and calculation of loss:

[0188] Enable gradient calculation, input data into the model for forward propagation, and obtain the log-likelihood value (logits). Calculate the loss function value based on the model output and the true label.

[0189] The loss function used in the machine learning model training in the embodiment of the present invention is: Binary Cross-Entropy (BCE) loss function, and the process of loading the loss function is:

[0190] First, get the positive sample weight: get the positive sample weight through train_dataset.getPosWeight() call. This weight is used to deal with the imbalance of positive and negative samples in the dataset, which can help the model better learn fewer categories. The weight calculation method is usually based on the ratio of the number of positive and negative samples.

[0191] We further place the positive sample weights on the CUDA device: we convert pos_weight to a PyTorch Tensor object and use the .cuda() method to ensure that it is transferred to the GPU. This is to maintain data compatibility with the model trained on the GPU.

[0192] Finally, set the loss function: initialize a binary cross entropy loss function object, torch.nn.BCEWithLogitsLoss(pos_weight=pos_weight). By passing the pos_weight parameter, we tell the loss function how to adjust the loss calculation based on the weight of the positive sample. BCEWithLogitsLoss internally combines the Sigmoid activation function and the calculation of the binary cross entropy loss, which means that the output of the model can be directly linear (without Sigmoid activation). The calculation formula of the binary cross entropy (BCE) loss function is as follows,

[0193]

[0194] Where L is the value of the loss function, N is the number of samples, and pos_weight is the weight given to the positive samples. i is the true label of the i-th sample, which is 1 if it is a positive class and 0 otherwise. i is the original output value (logits) of the model for the i-th sample, σ(x i ) is the sigmoid function applied to x i As a result, the original output value is mapped to the (0,1) interval, which is expressed as the probability of the positive class.

[0195] Step 506, back propagation and parameter update: Perform back propagation to calculate gradients. Update model weights based on the loss. If gradient clipping is set, apply it before parameter update to prevent gradient explosion.

[0196] Step 507, record and accumulate losses: accumulate the losses of this batch and update the total data volume. Collect model output and true labels separately for subsequent evaluation.

[0197] Step 508, Summarize and evaluate: After all batches of training are completed, all collected model outputs and labels are concatenated to form the outputs and labels of the entire training set. The computational complexity index of the model is printed, and the average loss of the entire epoch is calculated and printed. The evaluation function eval_func is called and the model output and true label of the entire training cycle are used as parameters for performance evaluation.

[0198] The evaluation indicators used in this embodiment include: accuracy (acc) and F1 score (F1),

[0199] In the binary classification problem, the confusion matrix is ​​shown in Table 1 (Confusion Matrix Table):

[0200] Table 1

[0201]

[0202] Among them, TP represents the number of positive samples predicted as positive; FP represents the number of negative samples predicted as positive; FN represents the number of positive samples predicted as negative; TN represents the number of negative samples predicted as negative.

[0203] The accuracy rate (acc) represents the ratio of the number of correctly predicted emotion categories TP+TN to the total number of emotion categories TP+TN+FP+FN in the total prediction results after model prediction. The calculation formula is as follows:

[0204]

[0205] It represents a comprehensive evaluation of precision (P) and recall (R). This value can more comprehensively evaluate the effect of the model. Its calculation formula is as follows:

[0206]

[0207] Among them, the calculation formulas of precision (Precision, P) and recall (Recall, R) are:

[0208]

[0209] The specific results are as follows: We selected several commonly used baseline models in multimodal emotion recognition and compared them with the best / most advanced models. We trained and evaluated them on the public datasets IEMOCAP and CMU-MOSEI. The results are shown in the following table and the line comparison chart is shown in the following table. Figure 4 , Figure 5 shown.

[0210] Table 2 IEMOCAP results display

[0211]

[0212] Table 3 CMU-MOSEI results display

[0213]

[0214] On the other hand, Figure 6 A multimodal emotion recognition device is provided, and the multimodal emotion recognition device is used to implement the multimodal emotion recognition method provided by an embodiment of the present invention, and the device includes:

[0215] The first acquisition module 601 is used to acquire text data of current emotion expression and construct a first text modality feature based on the text data using a pre-trained model;

[0216] The second acquisition module 602 is used to collect the voice data of the current emotion expression, and construct the first voice modality feature based on the voice data using a convolutional neural network, a bidirectional long short-term memory neural network and a pre-training model;

[0217] The third acquisition module 603 is used to acquire image data of current emotion expression, and construct a first image modality feature based on the image data using a preset convolutional network;

[0218] A fusion module 604 is configured to fuse the first text modality feature with the first image modality feature through a cross attention mechanism to generate an image feature with fused attention, and to fuse the first text modality feature with the first speech modality feature through a cross attention mechanism to generate a speech feature with fused attention;

[0219] The emotion recognition module 605 is used to send the speech features of the fused attention into the multi-layer Transformer encoder for learning to generate the second speech modality features, send the image features of the fused attention into the multi-layer Transformer encoder for learning to generate the second image modality features, send the first text modality features into the multi-layer Transformer encoder for learning to generate the second text modality features, and perform emotion recognition based on the second speech modality features, the second image modality features and the second text modality features.

[0220] On the other hand, a multimodal emotion recognition device is provided, the multimodal emotion recognition device comprising:

[0221] processor;

[0222] A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the method provided by the embodiment of the present invention.

[0223] On the other hand, a computer-readable storage medium is provided, in which a program code is stored. The program code can be called by a processor to execute the method provided by an embodiment of the present invention.

[0224] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0225] The embodiment of the present invention fuses the first text modal feature with the first image modal feature through a cross-attention mechanism to generate an image feature of fused attention, and fuses the first text modal feature with the first speech modal feature through a cross-attention mechanism to generate a speech feature of fused attention; the speech feature of fused attention is sent to a multi-layer Transformer encoder for learning to generate a second speech modal feature, the image feature of fused attention is sent to a multi-layer Transformer encoder for learning to generate a second image modal feature, the first text modal feature is sent to a multi-layer Transformer encoder for learning to generate a second text modal feature, and emotion recognition is performed based on the second speech modal feature, the second image modal feature and the second text modal feature, thereby achieving more accurate multimodal emotion recognition.

[0226] Figure 7 is a schematic diagram of the structure of a multimodal emotion recognition device provided by an embodiment of the present invention, such as Figure 7 As shown, optionally, the multimodal emotion recognition device 710 may include a first processor 2001.

[0227] Optionally, the multimodal emotion recognition device 710 may also include a memory 2002 and a transceiver 2003.

[0228] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0229] Combine the following Figure 7 The components of the multimodal emotion recognition device 710 are specifically introduced as follows:

[0230] The first processor 2001 is the control center of the multimodal emotion recognition device 710, and may be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or may be application specific integrated circuits (ASICs), or may be configured to implement one or more integrated circuits of the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0231] Optionally, the first processor 2001 can perform various functions of the multimodal emotion recognition device 710 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0232] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 7 CPU0 and CPU1 are shown in FIG.

[0233] In a specific implementation, as an embodiment, the multimodal emotion recognition device 710 may also include multiple processors, such as Figure 7 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0234] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0235] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 2001 through the interface circuit ( Figure 7 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0236] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0237] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 7 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0238] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and communicate with the first processor 2001 through the interface circuit ( Figure 7 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0239] It should be noted that Figure 7 The structure of the multimodal emotion recognition device 710 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0240] In addition, the technical effects of the multimodal emotion recognition device 710 can refer to the technical effects of the multimodal emotion recognition method described in the above method embodiment, and will not be repeated here.

[0241] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0242] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0243] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by infrared, wireless, microwave or the like. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0244] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0245] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0246] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0247] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0248] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0249] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0250] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0251] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0252] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0253] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multimodal emotion recognition method, characterized in that: The method comprises: Collecting text data of current emotional expression, and constructing a first text modality feature based on the text data using a pre-trained model; Collecting speech data of current emotional expression, and constructing a first speech modality feature based on the speech data using a convolutional neural network, a bidirectional long short-term memory neural network, and a pre-trained model; Collect image data of current emotional expression, and construct a first image modality feature based on the image data using a preset convolutional network; The first text modality feature is fused with the first image modality feature through a cross attention mechanism to generate an image feature of fused attention, and the first text modality feature is fused with the first speech modality feature through a cross attention mechanism to generate a speech feature of fused attention; The speech features with fused attention are sent to a multi-layer Transformer encoder for learning to generate a second speech modality feature, the image features with fused attention are sent to a multi-layer Transformer encoder for learning to generate a second image modality feature, the first text modality feature is sent to a multi-layer Transformer encoder for learning to generate a second text modality feature, and emotion recognition is performed based on the second speech modality feature, the second image modality feature and the second text modality feature.

2. The multimodal emotion recognition method according to claim 1, characterized in that: The method of using the pre-trained model Bert-base-multilingual-uncased-sentiment to construct the first text modality feature includes: First, use BERT's tokenizer to split the text into words or subwords; Add tags to the split text, convert the text into a vocabulary index, and generate an input tensor, wherein the input tensor includes a vocabulary index tensor, a position encoding tensor, and a segment encoding tensor; The input tensor is fed into the pre-trained Bert-base-multilingual-uncased-sentiment model, passed through a multi-layer Transformer encoder for contextual representation learning, and the labeled output vector is extracted as the first text modality feature.

3. The multimodal emotion recognition method according to claim 1, characterized in that: The method of constructing a first speech modality feature based on the speech data using a convolutional neural network, a bidirectional long short-term memory neural network and a pre-trained model includes: Read the speech data through three channels of spectrogram, MFCC and original audio data; Use the ResNet network to extract spectrogram features from the spectrogram, use the BiLSTM network to extract MFCC features from the MFCC data, and use the wav2vec 2.0 network to extract raw audio features from the raw audio data; Merging the spectrogram features with the MFCC features to generate merged features, and refining the merged features through a linear layer to generate weights of wav2vec 2.0 features in different frames, and weighting the original audio features according to the weights to generate weighted audio features; The spectrogram features, the MFCC features and the weighted audio features are connected to construct a comprehensive acoustic feature.

4. The multimodal emotion recognition method according to claim 1, characterized in that: The constructing a first image modality feature based on the image data using a preset convolutional network includes: extracting a frame image from the image data; Use MTCNN to identify facial information in frame images and locate facial key points; The frame image after locating the facial key points is sent to a preset convolutional network to extract the first image modality features. The preset convolutional network includes multiple convolutional layers, a RepVGG Block module, a batch normalization layer, an activation function layer and a maximum pooling layer.

5. The multimodal emotion recognition method according to claim 1, characterized in that: The method comprises: fusing the first text modality feature with the first image modality feature through a cross attention mechanism to generate an image feature with fused attention, and fusing the first text modality feature with the first speech modality feature through a cross attention mechanism to generate a speech feature with fused attention, including: Using the first image modality feature as a first query vector feature map, and the first text modality feature as a queried feature map; Perform cross attention calculation on the first query vector feature map and the queried feature map, select the first p parts of the probability in each attention score map by kernel sampling method and perform broadcast point-by-point multiplication with the original feature map to obtain image features of fused attention; The first speech modality feature is used as a second query vector feature graph, and the first text modality feature is used as a queried feature graph; Perform cross attention calculation on the second query vector feature map and the queried feature map, select the first p parts of the probability in each attention score map and perform broadcast point-by-point multiplication with the original feature map by the kernel sampling method to obtain speech features of fused attention; The P is a positive integer.

6. The multimodal emotion recognition method according to claim 1, characterized in that: Also includes: The machine learning model is trained through gradient calculation and batch normalization to implement the multimodal emotion recognition method using the machine learning model.

7. The multimodal emotion recognition method according to claim 6, characterized in that: The loss function of the machine learning model is a binary cross entropy loss function.

8. A multimodal emotion recognition device, the multimodal emotion recognition device is used to implement the multimodal emotion recognition method according to any one of claims 1 to 7, characterized in that: The device comprises: A first acquisition module, used for acquiring text data of current emotional expression, and constructing a first text modality feature based on the text data using a pre-trained model; A second acquisition module is used to collect voice data of current emotional expression, and construct a first voice modality feature based on the voice data using a convolutional neural network, a bidirectional long short-term memory neural network and a pre-trained model; A third acquisition module is used to acquire image data of current emotion expression, and construct a first image modality feature based on the image data using a preset convolutional network; A fusion module is configured to fuse the first text modality feature with the first image modality feature through a cross attention mechanism to generate an image feature of fused attention, and to fuse the first text modality feature with the first speech modality feature through a cross attention mechanism to generate a speech feature of fused attention; An emotion recognition module is used to send the speech features of the fused attention into a multi-layer Transformer encoder for learning to generate a second speech modality feature, send the image features of the fused attention into a multi-layer Transformer encoder for learning to generate a second image modality feature, send the first text modality feature into a multi-layer Transformer encoder for learning to generate a second text modality feature, and perform emotion recognition based on the second speech modality feature, the second image modality feature and the second text modality feature.

9. A multimodal emotion recognition device, characterized in that: The multimodal emotion recognition device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images