Multi-modal emotion recognition method and device

Abstract

The invention discloses a multi-modal emotion recognition method and a multi-modal emotion recognition device, which are used for solving the technical problem of poor recognition precision caused by the existing multi-modal emotion recognition method. The method comprises the steps that to-be-detected multi-modal emotion data are acquired, the to-be-detected emotion data are input into a preset multi-modal emotion recognition model, and the preset multi-modal emotion recognition model comprises a multi-modal feature extraction network, a cross-modal feature interaction network, a representation learning network and a feature modal recognition network; performing feature extraction on the to-be-detected multi-modal emotion data through the multi-modal feature extraction network to generate multi-modal emotion perception features; inputting the multi-modal emotion perception features into a cross-modal feature interaction network for feature interaction, and outputting multi-modal interaction features; performing feature learning on the multi-modal interaction features by adopting a representation learning network to generate a plurality of modal hidden vectors; and performing emotion recognition on the plurality of modal hidden vectors through a feature modal recognition network to generate a target multi-modal emotion recognition result.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal emotion recognition method and device. Background Art

[0002] Multimodal emotion recognition is one of the important research directions in emotion computing in recent years, which usually involves multimodal data such as text, audio and video. These modalities capture emotional information from different perspectives, but the amount and quality of information provided by each modality are different.

[0003] For single-modal sentiment analysis, although a preliminary judgment of sentiment can be obtained through text, audio or video modalities, this single-modal analysis often cannot fully capture the complexity and diversity of sentiment. Therefore, using multimodal information for fusion processing has become the mainstream trend in the current field of sentiment recognition.

[0004] Most of the existing multimodal emotion recognition methods construct corresponding deep networks for multimodal feature extraction and multimodal feature fusion. On this basis, the multimodal fusion features output by the deep network are sent to the corresponding classifier for multimodal emotion recognition. However, in the multimodal fusion process, it is difficult to effectively fuse cross-modal features, resulting in the loss of valuable emotional information or the increase of redundant information, leading to poor recognition accuracy. Summary of the invention

[0005] The present invention provides a multimodal emotion recognition method and device, which are used to solve the technical problem of poor recognition accuracy caused by the existing multimodal emotion recognition methods.

[0006] A first aspect of the present invention provides a multimodal emotion recognition method, comprising:

[0007] Acquire multimodal emotion data to be detected, and input the emotion data to be detected into a preset multimodal emotion recognition model, wherein the preset multimodal emotion recognition model includes a multimodal feature extraction network, a cross-modal feature interaction network, a representation learning network, and a feature modality recognition network;

[0008] Extracting features of the multimodal emotion data to be detected through the multimodal feature extraction network to generate multimodal emotion perception features;

[0009] Inputting the multimodal emotion perception features into the cross-modal feature interaction network for feature interaction, and outputting multimodal interaction features;

[0010] Using the representation learning network to perform feature learning on the multimodal interaction features to generate multiple modal hidden vectors;

[0011] Emotion recognition is performed on the multiple modal hidden vectors through the characteristic modal recognition network to generate a target multimodal emotion recognition result.

[0012] Optionally, the multimodal emotion data to be detected includes a text sequence to be detected, an audio to be detected, and a video frame image to be detected; the multimodal feature extraction network includes a language representation network, a recursive neural network, and a face detection network; the multimodal emotion perception features include text perception features, audio perception features, and video frame perception features; the multimodal emotion data to be detected is subjected to feature extraction through the multimodal feature extraction network to generate multimodal emotion perception features, including:

[0013] Extracting text features from the text sequence to be detected through a language representation network to generate text features;

[0014] Using a recursive neural network to extract audio features from the audio to be detected to generate audio features;

[0015] Inputting the video frame image to be detected into a face detection network to extract facial features and output a facial action unit sequence;

[0016] Modeling the temporal features of the facial action unit sequence through a recurrent neural network to generate video frame features;

[0017] Based on preset position embedding, the text features, the audio features and the video frame features are encoded respectively to generate text-perceptual features, audio-perceptual features and video frame-perceptual features.

[0018] Optionally, the cross-modal feature interaction network includes an activation function layer, a fully connected layer, and a bidirectional long short-term memory module; the multimodal interaction features include target text interaction features, target audio interaction features, and target video frame interaction features; the multimodal emotion perception features are input into the cross-modal feature interaction network for feature interaction, and the multimodal interaction features are output, including:

[0019] The text perception feature, the audio perception feature and the video frame perception feature are linearly transformed respectively through a fully connected layer to generate a text feature matrix, an audio feature matrix and a video frame feature matrix;

[0020] Generate initial text interaction features using an activation function layer according to a text query matrix in the text feature matrix, a video frame key value matrix and a video frame value matrix in the video frame feature matrix, and an audio key value matrix and an audio value matrix in the audio feature matrix;

[0021] Generate initial video frame interaction features according to a video frame query matrix in the video frame feature matrix, a text key value matrix and a text value matrix in the text feature matrix, and an audio key value matrix and an audio value matrix in the audio feature matrix using an activation function layer;

[0022] Generate initial audio interaction features using an activation function layer according to an audio query matrix in the audio feature matrix, a text key value matrix and a text value matrix in the text feature matrix, and a video frame key value matrix and a video frame value matrix in the video frame feature matrix;

[0023] The initial text interaction features, the initial video frame interaction features and the initial audio interaction features are respectively input into a bidirectional long short-term memory module for nonlinear mapping, and target text interaction features, target audio interaction features and target video frame interaction features are output.

[0024] Optionally, the activation function layer is used to generate initial text interaction features according to a text query matrix in the text feature matrix, a video frame key value matrix and a video frame value matrix in the video frame feature matrix, and an audio key value matrix and an audio value matrix in the audio feature matrix, including:

[0025] An activation function layer is used to calculate a text query matrix in the text feature matrix and a video frame key value matrix in the video frame feature matrix, and output a text-video-frame cross-modal attention matrix;

[0026] Performing a matrix multiplication operation on the text-video frame cross-modal attention matrix and the video frame value matrix in the video frame feature matrix, and outputting a first text interaction feature;

[0027] An activation function layer is used to calculate the text query matrix in the text feature matrix and the audio key value matrix in the audio feature matrix, and output a text-audio cross-modal attention matrix;

[0028] Performing a matrix multiplication operation on the text-audio cross-modal attention matrix and the audio value matrix in the audio feature matrix, and outputting a second text interaction feature;

[0029] The first text interaction feature and the second text interaction feature are concatenated to generate an initial text interaction feature.

[0030] Optionally, the using an activation function layer to generate initial video frame interaction features according to a video frame query matrix in the video frame feature matrix, a text key value matrix and a text value matrix in the text feature matrix, and an audio key value matrix and an audio value matrix in the audio feature matrix comprises:

[0031] An activation function layer is used to calculate the video frame query matrix in the video frame feature matrix and the text key value matrix in the text feature matrix, and output a video frame text cross-modal attention matrix;

[0032] Performing a matrix multiplication operation on the video frame text cross-modal attention matrix and the text value matrix in the text feature matrix, and outputting a first video frame interaction feature;

[0033] An activation function layer is used to calculate the video frame query matrix in the video frame feature matrix and the audio key value matrix in the audio feature matrix, and output a video frame audio cross-modal attention matrix;

[0034] Performing a matrix multiplication operation on the video frame audio cross-modal attention matrix and the audio value matrix in the audio feature matrix, and outputting a second video frame interaction feature;

[0035] The first video frame interaction feature and the second video frame interaction feature are spliced ​​to generate an initial video interaction feature.

[0036] Optionally, the using an activation function layer to generate initial audio interaction features according to an audio query matrix in the audio feature matrix, a text key value matrix and a text value matrix in the text feature matrix, and a video frame key value matrix and a video frame value matrix in the video frame feature matrix includes:

[0037] An activation function layer is used to calculate the audio query matrix in the audio feature matrix and the text key value matrix in the text feature matrix, and output an audio-text cross-modal attention matrix;

[0038] Performing a matrix multiplication operation on the audio-text cross-modal attention matrix and the text value matrix in the text feature matrix, and outputting a first audio interaction feature;

[0039] An activation function layer is used to calculate the audio query matrix in the audio feature matrix and the video frame key value matrix in the video frame feature matrix, and output an audio and video frame cross-modal attention matrix;

[0040] Performing a matrix multiplication operation on the audio-video frame cross-modal attention matrix and the video frame value matrix in the video frame feature matrix, and outputting a second audio interaction feature;

[0041] The first audio interaction feature and the second audio interaction feature are concatenated to generate an initial audio interaction feature.

[0042] Optionally, the representation learning network includes a first feedforward neural layer and a second feedforward neural layer; the multiple modal hidden vectors include multiple first modal hidden vectors and multiple second modal hidden vectors; and the method of using the representation learning network to perform feature learning on the multimodal interaction features to generate multiple modal hidden vectors includes:

[0043] Inputting the target text interaction feature, the target audio interaction feature and the target video frame interaction feature into a first feedforward neural layer for feature extraction, and outputting a plurality of first modal hidden vectors;

[0044] The target text interaction features, the target audio interaction features and the target video frame interaction features are respectively input into the second feedforward neural layer for feature extraction, and a plurality of second modality hidden vectors are output.

[0045] Optionally, the characteristic modality recognition network includes a multi-head self-attention mechanism layer and a prediction layer; the emotion recognition is performed on the multiple modality hidden vectors through the characteristic modality recognition network to generate a target multimodal emotion recognition result, including:

[0046] Stacking a plurality of the modal hidden vectors to generate a modal hidden vector matrix;

[0047] Using the modal hidden vector matrix as the input of the multi-head self-attention mechanism layer, and outputting the attention matrix;

[0048] Concatenate multiple hidden vectors in the attention matrix to generate a joint vector;

[0049] The prediction layer is used to predict the joint vector to generate a target multimodal emotion recognition result.

[0050] Optionally, the training process of the preset multimodal emotion recognition model includes:

[0051] Acquire multimodal emotion data to be trained, input the multimodal emotion data to be trained into an initial multimodal emotion recognition model, and output a plurality of first modality hidden vectors to be trained, a plurality of second modality hidden vectors to be trained, and a multimodal emotion recognition result to be trained;

[0052] Based on the back propagation algorithm, determining the model gradient according to the plurality of first modality hidden vectors to be trained, the plurality of second modality hidden vectors to be trained, and the multimodal emotion recognition results to be trained;

[0053] Using the model gradient to update the initial multimodal emotion recognition model to determine an intermediate multimodal emotion recognition model;

[0054] Calculating a target loss value according to the multimodal emotion data to be trained using the intermediate multimodal emotion recognition model and a preset loss function;

[0055] Determining whether the target loss value converges;

[0056] If converged, the intermediate multimodal emotion recognition model is used as the trained preset multimodal emotion recognition model.

[0057] A second aspect of the present invention provides a multimodal emotion recognition device, comprising:

[0058] An acquisition module, used for acquiring multimodal emotion data to be detected, and inputting the emotion data to be detected into a preset multimodal emotion recognition model, wherein the preset multimodal emotion recognition model includes a multimodal feature extraction network, a cross-modal feature interaction network, a representation learning network, and a feature modality recognition network;

[0059] A generating module, used for performing feature extraction on the multimodal emotion data to be detected through the multimodal feature extraction network to generate multimodal emotion perception features;

[0060] An output module, used for inputting the multimodal emotion perception features into the cross-modal feature interaction network for feature interaction, and outputting multimodal interaction features;

[0061] A learning module, used to perform feature learning on the multimodal interaction features using the representation learning network to generate multiple modal hidden vectors;

[0062] The recognition module is used to perform emotion recognition on the multiple modal hidden vectors through the characteristic modal recognition network to generate a target multimodal emotion recognition result.

[0063] It can be seen from the above technical solutions that the present invention has the following advantages:

[0064] The above technical scheme of the present invention provides a multimodal emotion recognition method, which first obtains multimodal emotion data to be detected, and inputs the emotion data to be detected into a preset multimodal emotion recognition model, which includes a multimodal feature extraction network, a cross-modal feature interaction network, a representation learning network and a feature modality recognition network; then, the multimodal emotion data to be detected is subjected to feature extraction through the multimodal feature extraction network to generate multimodal emotion perception features; the multimodal emotion perception features are input into the cross-modal feature interaction network for feature interaction, and multimodal interaction features are output; the multimodal interaction features are subjected to feature learning through the representation learning network to generate multiple modal hidden vectors; finally, the multiple modal hidden vectors are subjected to emotion recognition through the feature modality recognition network to generate a target multimodal emotion recognition result; based on the above scheme, the multimodal emotion perception features generated based on the multimodal feature extraction network are subjected to feature interaction through the cross-modal feature interaction network, and the multimodal interaction features are output. The representation learning network and the feature modal recognition network are combined to generate the target multimodal emotion recognition result process, and the rich multimodal emotion information is used to improve the feature expression ability, thereby enhancing the effectiveness of the feature in emotion expression, reducing the impact of noise, and thus improving the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0066] Figure 1 A flowchart of the steps of a multimodal emotion recognition method provided in Example 1 of the present invention;

[0067] Figure 2 A schematic diagram of the structure of the interactive feature processing principle provided in the first embodiment of the present invention;

[0068] Figure 3 A flowchart of the steps of the training process of the preset multimodal emotion recognition model provided in the second embodiment of the present invention;

[0069] Figure 4 A schematic diagram of a training framework for a preset multimodal emotion recognition model provided in Embodiment 2 of the present invention;

[0070] Figure 5 This is a structural block diagram of a multimodal emotion recognition device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0071] The embodiments of the present invention provide a multimodal emotion recognition method and device, which are used to solve the technical problem of poor recognition accuracy caused by existing multimodal emotion recognition methods.

[0072] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] See also Figure 1 , Figure 1 A flowchart of the steps of a multimodal emotion recognition method provided in Example 1 of the present invention.

[0074] The present invention provides a multimodal emotion recognition method, comprising:

[0075] Step 101: obtain multimodal emotion data to be detected, and input the emotion data to be detected into a preset multimodal emotion recognition model, wherein the preset multimodal emotion recognition model includes a multimodal feature extraction network, a cross-modal feature interaction network, a representation learning network, and a feature modality recognition network.

[0076] It should be noted that the data of the present invention comes from multimodal input, that is, the multimodal emotion data to be detected includes text, audio and video (text sequence to be detected, audio to be detected and video frame image to be detected).

[0077] Step 102: extract features from the multimodal emotion data to be detected through a multimodal feature extraction network to generate multimodal emotion perception features.

[0078] The multimodal feature extraction network includes a language representation network, a recurrent neural network, and a face detection network.

[0079] Multimodal emotion-aware features include text-aware features, audio-aware features, and video frame-aware features.

[0080] It should be noted that in order to ensure the effective extraction of information from each modality and provide high-quality input features for subsequent cross-modal feature fusion, the present invention adopts a language representation network (BERT model, Bidirectional Encoder Representations from Transformers) to process text, processes audio and video features based on a custom recursive neural network (sLSTM, Stacked Long Short-Term Memory) structure, and combines modality-invariant features with a cross-modal feature interaction network (Crossmodal Transformer) for modal fusion.

[0081] Specifically, the process of extracting features from the multimodal emotion data to be detected through the multimodal feature extraction network and generating multimodal emotion perception features can be achieved by executing the following sub-steps S11 to S15:

[0082] Step S21, extracting text features from the text sequence to be detected through a language representation network to generate text features;

[0083] The language representation network is the BERT model; BERT is a bidirectional pre-trained language model based on Transformer that can effectively capture emotional information in the context.

[0084] It should be noted that the text part uses the BERT model for feature extraction. The present invention uses the pre-trained BERT-base-uncased model to process the text. The text is processed by BERT's Tokenizer, and the length of the sentence sequence is padded or truncated, with a fixed length of 50. The process of extracting text features through BERT can be expressed as:

[0085] ;

[0086] in, is a text feature, and its feature dimension is [50,768]; It is the BERT model; is the text sequence to be detected.

[0087] Step S22: extracting audio features from the audio to be detected using a recursive neural network to generate audio features;

[0088] The audio to be detected is preliminarily processed by the Librosa tool and converted into audio data in a sequence format suitable for sLSTM processing.

[0089] The recursive neural network adopts the sLSTM structure, which is a recursive neural network specially designed for processing time series data and can effectively capture the emotional changes and time dependencies in audio sequences.

[0090] It should be noted that audio features are extracted through a custom sLSTM structure. Preliminary processing is performed through the Librosa tool to extract key audio features such as Mel spectrum and pitch, and convert them into a sequence format suitable for sLSTM processing to obtain the audio to be detected (audio sequence). The obtained audio sequence is input into sLSTM to generate audio features; among them, the sLSTM structure helps to reduce noise in the audio and enhance the emotional information expression of the audio modality. The processing process of audio features can be expressed as:

[0091] ;

[0092] in, is the audio feature, and its dimension is [T A ,d A ], the length of the audio sequence is T A is set to 500, and the feature dimension d A is 5; is the audio to be detected; is a recurrent neural network.

[0093] Step S23, inputting the video frame image to be detected into a face detection network to extract facial features, and outputting a facial action unit sequence;

[0094] It should be noted that the present invention uses the OpenFace tool (face detection network) to extract facial features in the video frame by frame, and extracts important facial action units from each frame of the video image through face detection and key point detection algorithms, thereby obtaining a facial action unit sequence.

[0095] Step S24, performing temporal feature modeling on the facial action unit sequence through a recursive neural network to generate video frame features;

[0096] It should be noted that the extracted frame sequence (facial action unit sequence) is input into sLSTM for temporal feature modeling to generate video frame features. Among them, sLSTM models the temporal features of the frame sequence, which improves the performance of the video modality in emotion recognition. The processing process of video frame features can be expressed as:

[0097] ;

[0098] in, is the video frame feature, The dimension is [T V ,d V], the frame length of the video sequence is T V is set to 375, and the feature dimension d of each frame V is 20; is the sequence of facial action units; is a recurrent neural network.

[0099] Step S25: Based on the preset position embedding, the text features, audio features and video frame features are encoded respectively to generate text-perceptual features, audio-perceptual features and video frame-perceptual features.

[0100] It should be noted that, in order to enable the sequence to carry time information, the present invention enhances the position embedding (PE), that is, the preset position embedding can be expressed as: ; The processing process of text perception features, audio perception features and video frame perception features can be expressed as:

[0101] ;

[0102] in, It is a multimodal emotion-aware feature, including text-aware features, audio-aware features, and video frame-aware features, which represent low-level position-aware features for different modes; Embed for preset position; is the embedding indexed at each position, ; is a multimodal feature, including text features, audio features, and video frame features; d is the embedding dimension; is the sequence length of the multimodal emotion perception features.

[0103] Step 103: input the multimodal emotion perception features into a cross-modal feature interaction network for feature interaction, and output multimodal interaction features.

[0104] The cross-modal feature interaction network includes an activation function layer, a fully connected layer, and a bidirectional long short-term memory module; the activation function layer adopts a Softmax activation function; and the bidirectional long short-term memory module adopts a stacked bidirectional long short-term memory network.

[0105] The multimodal interaction features include target text interaction features, target audio interaction features and target video frame interaction features.

[0106] Specifically, the process of inputting the multimodal emotion perception features into the cross-modal feature interaction network for feature interaction and outputting the multimodal interaction features can be achieved by executing the following sub-steps S31 to S35:

[0107] Step S31, linearly transforming the text perception features, the audio perception features, and the video frame perception features through a fully connected layer to generate a text feature matrix, an audio feature matrix, and a video frame feature matrix;

[0108] The text feature matrix includes the text query matrix, the text key value matrix and the text value matrix.

[0109] The audio feature matrix includes an audio query matrix, an audio key-value matrix, and an audio value matrix.

[0110] The video frame feature matrix includes a video frame query matrix, a video frame key value matrix and a video frame value matrix.

[0111] Step S32, using an activation function layer to generate initial text interaction features according to a text query matrix in the text feature matrix, a video frame key value matrix in the video frame feature matrix, a video frame value matrix, and an audio key value matrix and an audio value matrix in the audio feature matrix;

[0112] Specifically, step S32 may include the following sub-steps S321-S325:

[0113] S321, using an activation function layer to calculate the text query matrix in the text feature matrix and the video frame key value matrix in the video frame feature matrix, and output a text video frame cross-modal attention matrix;

[0114] S322, performing a matrix multiplication operation on the text video frame cross-modal attention matrix and the video frame value matrix in the video frame feature matrix, and outputting a first text interaction feature;

[0115] S323, using an activation function layer to calculate the text query matrix in the text feature matrix and the audio key value matrix in the audio feature matrix, and outputting a text-audio cross-modal attention matrix;

[0116] S324, performing a matrix multiplication operation on the text-audio cross-modal attention matrix and the audio value matrix in the audio feature matrix, and outputting a second text interaction feature;

[0117] S325: Concatenate the first text interaction feature and the second text interaction feature to generate an initial text interaction feature.

[0118] It should be noted that the present invention considers two modes: and , represented by two (possibly unaligned) sequences from them and ,in, For modal The sequence length, For modal The characteristic dimension of For modal The sequence length, For modal Inspired by the decoder Transformer in NMT that translates one language into another, the present invention assumes that a good way to fuse cross-modal information is to provide cross-modal potential adaptation, that is, arrive .

[0119] Furthermore, the present invention defines the query matrix as , the key value matrix is , the value matrix is ,in, , , , is the weight corresponding to the query matrix, is the weight corresponding to the key-value matrix, is the weight corresponding to the value matrix, d k is the scaling factor, d v is the dimension of the value matrix. arrive The potential adaptation of is presented through cross-modal attention, that is, the processing principle of interaction features (including the first interaction matrix and the second interaction matrix, the first interaction matrix includes the first text interaction matrix, the first audio interaction matrix and the first video frame interaction matrix; the second interaction matrix includes the second text interaction matrix, the second audio interaction matrix and the second video frame interaction matrix), which can be expressed as:

[0120] ;

[0121] in, is the interactive feature, ; for cross-modal attention operations; For modal ; For modal ; is the Softmax activation function; is the query matrix; is the scaling factor; For modal The corresponding value matrix; T is the transpose; is the weight corresponding to the query matrix; is the weight corresponding to the key-value matrix; is the weight corresponding to the value matrix; for The transposed matrix of is the dimension of the value matrix.

[0122] It is worth mentioning that and have the same length (i.e. ), but at the same time Specifically, according to The scaled Softmax calculates a score matrix , whose (i,j)th item represents the mode The i-th time step of the mode The attention of the jth time step. Therefore, The i-th time step of A weighted summary of The i-th row in is determined.

[0123] For further information, see Figure 2 , taking text-aware features as modality , video frame-aware features as modality When the text query matrix in the text feature matrix and the video frame key value matrix in the video frame feature matrix are calculated using the activation function layer, the text video frame cross-modal attention matrix is ​​output, the matrix multiplication operation is performed on the text video frame cross-modal attention matrix and the video frame value matrix in the video frame feature matrix, and the first text interaction feature is output; wherein, the processing process of the first text interaction feature can refer to the processing principle of the interaction feature, and the present invention will not go into details.

[0124] Next, we use the text-aware features as the modality , audio perception features as modality When the text query matrix in the text feature matrix and the audio key value matrix in the audio feature matrix are calculated using an activation function layer, the text audio cross-modal attention matrix is ​​output; the text audio cross-modal attention matrix and the audio value matrix in the audio feature matrix are subjected to matrix multiplication operation, and the second text interaction feature is output; wherein, the processing process of the second text interaction feature can refer to the processing principle of the interaction feature, and the present invention will not go into details.

[0125] Finally, the first text interaction feature and the second text interaction feature are concatenated to generate the initial text interaction feature. The processing principle of the initial interaction feature (including the initial text interaction feature, the initial audio interaction feature and the initial video frame interaction feature) can be expressed as:

[0126] ;

[0127] in, is the initial interaction feature, , l represents text, v represents video frame, and a represents audio; For splicing; is a first interaction feature, including a first text interaction matrix, a first audio interaction matrix, and a first video frame interaction matrix; The second interaction feature includes a second text interaction matrix, a second audio interaction matrix and a second video frame interaction matrix.

[0128] In this embodiment, the present invention brings the text, video and audio features output by the feature extraction part into the modality and modality splicing respectively, so as to integrate the information of two different modalities into the third modality respectively, and performs splicing operation on the two interactive features obtained, thereby outputting the feature representation in the group.

[0129] Step S33, using an activation function layer to generate initial video frame interaction features according to the video frame query matrix in the video frame feature matrix, the text key value matrix and the text value matrix in the text feature matrix, and the audio key value matrix and the audio value matrix in the audio feature matrix;

[0130] Specifically, step S33 may include the following sub-steps S331-S335:

[0131] S331, using an activation function layer to calculate the video frame query matrix in the video frame feature matrix and the text key value matrix in the text feature matrix, and outputting a video frame text cross-modal attention matrix;

[0132] S332, performing matrix multiplication operation on the video frame text cross-modal attention matrix and the text value matrix in the text feature matrix, and outputting the first video frame interaction feature;

[0133] S333, using an activation function layer to calculate the video frame query matrix in the video frame feature matrix and the audio key value matrix in the audio feature matrix, and outputting a video frame audio cross-modal attention matrix;

[0134] S334, performing matrix multiplication operation on the audio value matrix in the video frame audio cross-modal attention matrix and the audio feature matrix, and outputting the second video frame interaction feature;

[0135] S335 . Concatenate the first video frame interaction feature and the second video frame interaction feature to generate an initial video interaction feature.

[0136] It should be noted that the principle of the steps of the initial video interaction feature is consistent with the principle of the steps of the above-mentioned initial text interaction feature, and the present invention will not be repeated in detail.

[0137] Step S34, using an activation function layer to generate initial audio interaction features according to the audio query matrix in the audio feature matrix, the text key value matrix and the text value matrix in the text feature matrix, and the video frame key value matrix and the video frame value matrix in the video frame feature matrix;

[0138] Specifically, step S34 may include the following sub-steps S341-S345:

[0139] S341, using an activation function layer to calculate the audio query matrix in the audio feature matrix and the text key value matrix in the text feature matrix, and outputting an audio-text cross-modal attention matrix;

[0140] S342, performing a matrix multiplication operation on the audio text cross-modal attention matrix and the text value matrix in the text feature matrix, and outputting a first audio interaction feature;

[0141] S343, using an activation function layer to calculate the audio query matrix in the audio feature matrix and the video frame key value matrix in the video frame feature matrix, and outputting an audio and video frame cross-modal attention matrix;

[0142] S344, performing a matrix multiplication operation on the audio and video frame cross-modal attention matrix and the video frame value matrix in the video frame feature matrix, and outputting a second audio interaction feature;

[0143] S345: Concatenate the first audio interaction feature and the second audio interaction feature to generate an initial audio interaction feature.

[0144] It should be noted that the principle of the steps of the initial audio interaction feature is consistent with the principle of the steps of the above-mentioned initial text interaction feature, and the present invention will not be elaborated on.

[0145] Step S35: input the initial text interaction features, the initial video frame interaction features and the initial audio interaction features into the bidirectional long short-term memory module for nonlinear mapping, and output the target text interaction features, the target audio interaction features and the target video frame interaction features.

[0146] It should be noted that the present invention converts the speech sequence, that is, the initial interaction feature Mapping to a fixed-size vector The present invention uses a stacked bidirectional long short-term memory network (LSTM), i.e., a bidirectional long short-term memory module, whose hidden representation of the final state is combined with a fully connected dense layer to obtain ; Among them, the processing process of the target interaction features (including the target text interaction features, the target audio interaction features and the target video frame interaction features) can be expressed as:

[0147] ;

[0148] in, is the target interaction feature; It is a stacked bidirectional long short-term memory network; The network parameters corresponding to the stacked bidirectional long short-term memory network; is the initial interaction feature, , l represents text, v represents video frame, and a represents audio.

[0149] Step 104: Use a representation learning network to perform feature learning on multimodal interaction features to generate multiple modal hidden vectors.

[0150] The representation learning network includes a first feed-forward neural layer and a second feed-forward neural layer.

[0151] The multiple modal hidden vectors include multiple first modal hidden vectors and multiple second modal hidden vectors; the multiple first modal hidden vectors include a first text modal hidden vector, a first audio modal hidden vector and a first video frame modal hidden vector, and the multiple second modal hidden vectors include a second text modal hidden vector, a second audio modal hidden vector and a second video frame modal hidden vector.

[0152] Specifically, the process of using a representation learning network to perform feature learning on multimodal interaction features and generate multiple modal hidden vectors can be achieved by executing the following sub-steps S41 to S42:

[0153] Step S41, inputting the target text interaction features, the target audio interaction features and the target video frame interaction features into the first feedforward neural layer for feature extraction, and outputting a plurality of first modal hidden vectors;

[0154] It should be noted that the present invention converts each speech vector The projected representations are divided into two different representations, namely modality-invariant and modality-specific. The first is the modality-invariant component, which learns a shared representation in a common subspace with a distribution similarity constraint. This constraint helps minimize the heterogeneity gap, which is a desirable property for multimodal fusion and is expressed as The second is a modality-specific component that captures the unique features of that modality, resulting in a modality-specific hidden representation .

[0155] Furthermore, the processing process of the first modal hidden vector can be expressed as:

[0156] ;

[0157] in, is the first modal hidden vector; is the first feed-forward neural layer; is the target interaction feature; To share parameters among all three modalities.

[0158] Step S42: input the target text interaction features, the target audio interaction features and the target video frame interaction features into the second feedforward neural layer for feature extraction, and output a plurality of second modality hidden vectors.

[0159] It should be noted that the processing process of the second modal hidden vector can be expressed as:

[0160] ;

[0161] in, is the second modality hidden vector; is the second feed-forward neural layer; is the target interaction feature; Assign independent parameters to each mode.

[0162] In this embodiment, the present invention generates six hidden vectors , i.e., a first text modality hidden vector, a first audio modality hidden vector and a first video frame modality hidden vector, a second text modality hidden vector, a second audio modality hidden vector and a second video frame modality hidden vector, the present invention uses a simple first feedforward neural layer and the second feed-forward neural layer ; indicates that parameters are shared among all three modes, Indicates that independent parameters are assigned to each mode.

[0163] Step 105: Perform emotion recognition on multiple modal hidden vectors through a feature modal recognition network to generate a target multimodal emotion recognition result.

[0164] The feature modality recognition network includes a multi-head self-attention mechanism layer and a prediction layer.

[0165] It should be noted that after projecting the modalities to their respective representations, the present invention fuses them into a joint vector for downstream prediction. The present invention designs a simple fusion mechanism that first performs a self-attention mechanism and then connects all six transformed modality vectors.

[0166] Specifically, step 105 may include the following sub-steps S51-S54:

[0167] S51, stacking multiple modal hidden vectors to generate a modal hidden vector matrix;

[0168] It should be noted that the present invention stacks the six modal representations into a matrix, namely the modal latent vector matrix , d h is the hidden vector dimension.

[0169] S52, taking the modal hidden vector matrix as the input of the multi-head self-attention mechanism layer, and outputting the attention matrix;

[0170] It should be noted that the multi-head self-attention mechanism layer (Transformer) uses an attention module defined as a scaled dot product function, which is expressed as:

[0171] ;

[0172] in, is the attention module; Q is the query matrix; K is the key-value matrix; V is the value matrix; is the softmax activation function; is the scaling factor; T is the transpose.

[0173] Furthermore, the Transformer computes multiple such attentions in parallel, where each attention output is called a head. The head is calculated as follows:

[0174] ;

[0175] in, is the i-th head attention; The first head-specific parameter of the linear projection of the matrix corresponding to the i-th head attention into the local space; The second head-specific parameters of the linear projection of the matrix corresponding to the i-th head attention into the local space; is the third head-specific parameter of the linear projection of the matrix corresponding to the i-th head attention to the local space, , , ; is the attention module; Q is the query matrix; K is the key-value matrix; V is the value matrix.

[0176] Furthermore, the present invention performs a multi-head self-attention mechanism on the representation in the state hidden vector matrix, so that each vector can perceive other cross-modal (and cross-subspace) representations; for the self-attention mechanism, the present invention sets , a new matrix, namely the attention matrix, is generated through the multi-head self-attention mechanism layer. The processing process of the attention matrix can be expressed as:

[0177] ;

[0178] in, is the attention matrix; is a multi-head self-attention mechanism layer; M is the modal hidden vector matrix; is the weight vector, , is the first head-specific parameter for the linear projection of the matrix into local space, is the second head-specific parameter for the linear projection of the matrix into local space, is the third head-specific parameter for the linear projection of the matrix into local space, is the output weight matrix; For the first head attention; is the nth head attention.

[0179] S53, concatenating multiple hidden vectors in the attention matrix to generate a joint vector;

[0180] The multiple hidden vectors include a first text attention hidden vector, a first audio attention hidden vector, a first video frame attention hidden vector, a second text attention hidden vector, a second audio attention hidden vector, and a second video frame attention hidden vector.

[0181] It should be noted that the present invention takes the output of the last hidden layer of Transformer, that is, concatenates multiple hidden vectors in the attention matrix to construct a joint vector. The process can be expressed as:

[0182] ;

[0183] in, is the joint vector, ; is the first text attention hidden vector; is the attention hidden vector for the first video frame; is the first audio attention hidden vector; Hidden vector for the second text attention; Attention hidden vector for the second video frame; Hidden vector for the second audio attention; For splicing.

[0184] S54. Use the prediction layer to predict the joint vector and generate the target multimodal emotion recognition result.

[0185] It should be noted that the prediction process can be expressed as:

[0186] ;

[0187] in, The target multimodal emotion recognition result; is the joint vector; is a set of parameters; is the prediction function.

[0188] Furthermore, the scores measured on the data sets processed by the present invention are shown in Table 1. After experimental verification, the present invention has achieved certain effects; among them, the TFN, LMF, MFN, MULT, and MISA data in Table 1 are from Mao et al. (2022).

[0189] Table 1 Experimental verification

[0190]

[0191] As a comparison of technical effects, we can refer to existing technologies. Multimodal emotion recognition is one of the important directions in emotion computing research in recent years, which usually involves multimodal data such as text, audio and video. These modalities capture emotional information from different angles, but the amount and quality of information provided by each modality are different. For single-modal emotion analysis, although preliminary judgments of emotions can be obtained through text, audio or video modalities, such single-modal analysis often cannot fully capture the complexity and diversity of emotions. Therefore, the use of multimodal information for fusion processing has become the mainstream trend in the current field of emotion recognition.

[0192] However, there are still several key challenges in the multimodal emotion recognition task: (1) Information imbalance between different modalities: text usually contains richer emotional information, while the features of audio and video modalities may be sparse and contain more noise; (2) In the multimodal fusion process, it is difficult to effectively fuse cross-modal features, which may lead to the loss of valuable emotional information or the increase of redundant information; (3) General emotion recognition has a high discrimination for binary classification, but often ignores the accuracy of five-category and seven-category emotion recognition.

[0193] In response to the above problems, some studies have attempted to improve the performance of multimodal emotion recognition through deep learning technology, but there is still room for improvement in cross-modal feature fusion. Therefore, the present invention proposes a multimodal emotion recognition method, which introduces a cross-attention mechanism and utilizes the rich emotional information in the text modality to enhance the feature expression capabilities of the audio and video modalities. In this mechanism, text features interact with audio and video features, allowing the audio and video modalities to obtain valuable information from the text modality, thereby enhancing their effectiveness in emotional expression and reducing the impact of noise. At the same time, in order to prevent the loss of important emotional information during the modal interaction process, the present invention fuses the audio and video features after the interaction with the original features before the interaction, ensuring that valuable emotional information is retained during the feature fusion process, thereby improving the accuracy of multi-classification emotion recognition; in addition, the present invention addresses the problems of modal imbalance and information loss in multimodal emotion recognition. The cross-modal feature fusion process proposed in the present invention significantly improves the performance of multimodal emotion recognition, especially in the accuracy of five-category and seven-category emotion recognition tasks. By rationally utilizing the cross-attention mechanism and modality-invariant representation in the information fusion of different modalities, it can have significant advantages in multimodal emotion analysis tasks.

[0194] In an embodiment of the present invention, the present invention provides a multimodal emotion recognition method, which first obtains multimodal emotion data to be detected, and inputs the emotion data to be detected into a preset multimodal emotion recognition model, wherein the preset multimodal emotion recognition model includes a multimodal feature extraction network, a cross-modal feature interaction network, a representation learning network, and a feature modality recognition network; then, feature extraction is performed on the multimodal emotion data to be detected through the multimodal feature extraction network to generate multimodal emotion perception features; the multimodal emotion perception features are input into the cross-modal feature interaction network for feature interaction, and multimodal interaction features are output; feature learning is performed on the multimodal interaction features using the representation learning network to generate multiple modal hidden vectors; finally, emotion recognition is performed on the multiple modal hidden vectors through the feature modality recognition network to generate a target multimodal emotion recognition result; based on the above scheme, feature interaction is performed on the multimodal emotion perception features generated based on the multimodal feature extraction network through the cross-modal feature interaction network, and multimodal interaction features are output. The representation learning network and the feature modal recognition network are combined to generate the target multimodal emotion recognition result process, and the rich multimodal emotion information is used to improve the feature expression ability, thereby enhancing the effectiveness of the feature in emotion expression, reducing the impact of noise, and thus improving the recognition accuracy.

[0195] For better explanation, refer to Figure 3 , shows a flowchart of the steps of the training process of the preset multimodal emotion recognition model provided by the second embodiment of the present invention, and the process may include the following steps:

[0196] Step 301, obtain multimodal emotion data to be trained, input the multimodal emotion data to be trained into an initial multimodal emotion recognition model, and output a plurality of first modality hidden vectors to be trained, a plurality of second modality hidden vectors to be trained, and a multimodal emotion recognition result to be trained.

[0197] The multimodal emotion data to be trained includes text sequences to be trained, audio to be trained, and video frame images to be trained.

[0198] Please note that Figure 4 The training framework of the initial multimodal emotion recognition model proposed in the present invention is divided into five parts: feature extraction, cross-modality (Transformer), representation learning, decoder, and fusion module; wherein, the feature extraction, cross-modality, representation learning, and fusion modules respectively process the input multimodal emotion data to be trained through the initial multimodal emotion recognition model, and output the corresponding multimodal emotion perception features to be trained, multimodal interaction features to be trained, multiple first modal hidden vectors to be trained, multiple second modal hidden vectors to be trained, and multimodal emotion recognition results to be trained; the decoder reconstructs multiple first modal hidden vectors to be trained and multiple second modal hidden vectors to be trained through a decoder function, and determines the reconstructed modal vector.

[0199] Step 302: Based on the back propagation algorithm, determine the model gradient according to multiple first modality hidden vectors to be trained, multiple second modality hidden vectors to be trained, and multimodal emotion recognition results to be trained.

[0200] The multiple first modality hidden vectors to be trained include a first text modality hidden vector to be trained, a first audio modality hidden vector to be trained, and a first video frame modality hidden vector to be trained; the multiple second modality hidden vectors to be trained include a second text modality hidden vector to be trained, a second audio modality hidden vector to be trained, and a second video frame modality hidden vector to be trained.

[0201] Specifically, a preset similarity loss function is used to calculate a similarity loss value based on multiple first modal hidden vectors to be trained; a preset difference loss function is used to calculate a difference loss value based on multiple first modal hidden vectors to be trained and multiple second modal hidden vectors to be trained; vector reconstruction is performed on multiple first modal hidden vectors to be trained and multiple second modal hidden vectors to be trained to determine the reconstructed modal vector; the reconstructed modal vector is substituted into the preset reconstruction loss function for calculation to determine the reconstruction loss value; a preset task loss function is used to calculate the task loss value based on the multimodal emotion recognition result to be trained; a target loss value is determined based on the task loss value, the similarity loss value, the reconstruction loss value and the difference loss value; based on the back propagation algorithm, the model gradient is determined based on the target loss value.

[0202] Furthermore, minimizing the similarity loss can reduce the difference between the shared representations of each modality. This helps align common features across modalities in a shared subspace. The present invention uses "central moment difference" (CMD) as a metric to measure this difference. CMD is an advanced distance metric that measures the distribution difference of two representations by matching moment differences of different orders. Intuitively, the CMD distance decreases as the two distributions become more similar.

[0203] Furthermore, let X and Y be bounded random samples with probability distributions p and q, respectively, defined in the interval The regularization term of CMD is defined as CMD K Empirical estimates of indicators, specifically:

[0204] ;

[0205] in, It is the distance metric of the kth order moment of the central moment difference, which is used to measure the distribution difference of two samples X and Y in high-order features; X is the first sample; Y is the second sample; [a, b] is the range of samples X and Y; , x is the data point in sample X; is the empirical expectation vector of the first sample X; is the empirical expectation vector of the second sample Y; is the L2 norm; K is the center matrix order; is the kth order center distance vector of the first sample X, The feature description of the data distribution in the k-th order matrix is ​​used to capture the information of sample distribution on higher-order features; is the k-th center distance vector of the second sample Y;

[0206] Based on the above foundation, the first text modality hidden vector to be trained, the first audio modality hidden vector to be trained, and the first video frame modality hidden vector to be trained are paired and input into the preset similarity loss function for calculation, and the similarity loss value is output, where the preset similarity loss function can be expressed as:

[0207] ;

[0208] in, is the similarity loss value; The subscript is The first modal hidden vector of ; is the first modal hidden vector with subscript m2; is the central matrix difference metric function.

[0209] It is worth mentioning that there are two important observations in this paper: (i) CMD is chosen instead of KL divergence or MMD because CMD is a popular metric and it can explicitly match high-order moments without expensive distance and kernel matrix calculations. (ii) Adversarial loss is also another option for similarity training, using a discriminator and a shared encoder to play a min-max game. However, CMD is chosen because of its simple formulation. In contrast, adversarial training requires additional discriminator parameters and adds complications such as oscillations in training.

[0210] Furthermore, a discrepancy loss is used to ensure that the modality-invariant representation and the modality-specific representation capture different aspects of the input. Non-redundancy can be achieved by imposing a soft orthogonal constraint between the two representations. In a training batch of sentences, let and are matrices whose rows represent the hidden vectors of modality m. and ,in Then the orthogonality constraint of this modal vector pair is calculated by the following formula:

[0211] ;

[0212] in, is the second modality hidden vector matrix; is the squared Frobenius norm; is the first modal hidden vector matrix.

[0213] Based on the above foundation, in addition to the constraints between the invariant and specific vectors, orthogonal constraints are added between the modality-specific vectors. The first text modality hidden vector to be trained and the second text modality hidden vector to be trained, the first audio modality hidden vector to be trained and the second audio modality hidden vector to be trained, the first video frame modality hidden vector to be trained and the second video frame modality hidden vector to be trained are paired in pairs, and the second text modality hidden vector to be trained, the second audio modality hidden vector to be trained and the second video frame modality hidden vector to be trained are paired in pairs, and are input into the preset difference term loss function together to calculate the difference term loss value; wherein, the preset difference term loss function can be expressed as:

[0214] ;

[0215] in, is the second modality hidden vector matrix; is the squared Frobenius norm; is the first modal hidden vector matrix; is the difference loss value.

[0216] Furthermore, due to the application of the difference loss, the modality-specific encoder may learn irrelevant representations. This may occur if the encoder function approximates a vector that is orthogonal but cannot represent the modality. To avoid this situation, the present invention adds a reconstruction loss to ensure that the hidden representations can capture the details of their respective modalities. The modality vector is reconstructed by using a decoder function, that is, the first text modality hidden vector to be trained, the first audio modality hidden vector to be trained, the first video frame modality hidden vector to be trained, the second text modality hidden vector to be trained, the second audio modality hidden vector to be trained, and the second video frame modality hidden vector to be trained are vector reconstructed to determine the reconstructed modality vector; the decoder function can be expressed as:

[0217] ;

[0218] in, To reconstruct the modal vector; is the decoder function; is the first modality hidden vector to be trained; is the hidden vector of the second modality to be trained; is the parameter set of decoder D.

[0219] Furthermore, the reconstruction loss is the true modal vector and its reconstruction vector Therefore, the preset reconstruction loss function can be expressed as:

[0220] ;

[0221] in, is the reconstruction loss value; is the true mode vector; To reconstruct the modal vector; is the square of the L2 norm.

[0222] Furthermore, task-specific losses are used to evaluate the quality of predictions during training. For classification tasks, the present invention uses the standard cross entropy loss, while for regression tasks, the present invention uses the mean square error loss. Therefore, the preset task loss function can be expressed as:

[0223] ;

[0224] in, is the loss value of the first task; is the number of multimodal emotion recognition results to be trained; is the true label; The multimodal emotion recognition results to be trained; is the loss value of the second task; is the task loss value; is the square of the L2 norm.

[0225] Furthermore, the target loss value is determined according to the task loss value, similarity loss value, reconstruction loss value and difference loss value, and the model gradient is determined according to the target loss value based on the back propagation algorithm. The calculation process of the target loss value can be expressed as:

[0226] ;

[0227] Where L is the target loss value; is the task loss value; is the first interaction weight; is the second interaction weight; is the third interaction weight; is the similarity loss value; is the difference loss value; is the reconstruction loss value.

[0228] It is worth mentioning that the interaction weights determine the contribution of each regularization part to the overall loss L, and the purpose of each component loss is to achieve the desired subspace properties.

[0229] Step 303: Use the model gradient to update the initial multimodal emotion recognition model to determine the intermediate multimodal emotion recognition model.

[0230] Step 304: Use the intermediate multimodal emotion recognition model and the preset loss function to calculate the target loss value according to the multimodal emotion data to be trained.

[0231] The preset loss functions include preset similarity loss function, preset difference loss function, preset reconstruction loss function, and preset task loss function.

[0232] It should be noted that the intermediate multimodal emotion recognition model is used to obtain a new plurality of first modal hidden vectors to be trained, a new plurality of second modal hidden vectors to be trained, and a new multimodal emotion recognition result to be trained according to the multimodal emotion data to be trained, and then a preset loss function is used to calculate the target loss value according to the new plurality of first modal hidden vectors to be trained, the new plurality of second modal hidden vectors to be trained, and the new multimodal emotion recognition result to be trained.

[0233] Step 305: Determine whether the target loss value converges.

[0234] Step 306: If converged, the intermediate multimodal emotion recognition model is used as the trained preset multimodal emotion recognition model.

[0235] It should be noted that if the target loss value does not converge, the model gradient is determined based on the back propagation algorithm according to the target loss value, and the intermediate multimodal emotion recognition model is updated using the model gradient to determine a new intermediate multimodal emotion recognition model. The new intermediate multimodal emotion recognition model and the preset loss function are used to recalculate the target loss value according to the multimodal emotion data to be trained until the target loss value converges. When the target loss value converges, the intermediate multimodal emotion recognition model is determined as the trained preset multimodal emotion recognition model.

[0236] See also Figure 5 , Figure 5 This is a structural block diagram of a multimodal emotion recognition device provided in Example 3 of the present invention.

[0237] The present invention provides a multimodal emotion recognition device, comprising:

[0238] An acquisition module 501 is used to acquire multimodal emotion data to be detected, and input the emotion data to be detected into a preset multimodal emotion recognition model, wherein the preset multimodal emotion recognition model includes a multimodal feature extraction network, a cross-modal feature interaction network, a representation learning network, and a feature modality recognition network;

[0239] A generating module 502 is used to extract features from the multimodal emotion data to be detected through a multimodal feature extraction network to generate multimodal emotion perception features;

[0240] An output module 503 is used to input the multimodal emotion perception features into a cross-modal feature interaction network for feature interaction, and output multimodal interaction features;

[0241] A learning module 504 is used to perform feature learning on multimodal interaction features using a representation learning network to generate multiple modal hidden vectors;

[0242] The recognition module 505 is used to perform emotion recognition on multiple modal hidden vectors through a feature modal recognition network to generate a target multi-modal emotion recognition result.

[0243] Further, the multimodal emotion data to be detected includes a text sequence to be detected, an audio to be detected, and a video frame image to be detected; the multimodal feature extraction network includes a language representation network, a recursive neural network, and a face detection network; the multimodal emotion perception features include text perception features, audio perception features, and video frame perception features; the generation module 502 is specifically used to:

[0244] The text features of the text sequence to be detected are extracted through the language representation network to generate text features;

[0245] A recursive neural network is used to extract audio features from the audio to be detected and generate audio features;

[0246] Input the video frame image to be detected into the face detection network to extract facial features and output a facial action unit sequence;

[0247] The temporal features of facial action unit sequences are modeled through a recurrent neural network to generate video frame features.

[0248] Based on preset position embedding, text features, audio features and video frame features are encoded respectively to generate text-aware features, audio-aware features and video frame-aware features.

[0249] Furthermore, the cross-modal feature interaction network includes an activation function layer, a fully connected layer, and a bidirectional long short-term memory module; the multimodal interaction features include target text interaction features, target audio interaction features, and target video frame interaction features; the output module 503 includes:

[0250] The first submodule is used to perform linear transformation on the text perception features, the audio perception features and the video frame perception features through a fully connected layer to generate a text feature matrix, an audio feature matrix and a video frame feature matrix;

[0251] The second submodule is used to generate initial text interaction features according to the text query matrix in the text feature matrix, the video frame key value matrix in the video frame feature matrix, the video frame value matrix, and the audio key value matrix and the audio value matrix in the audio feature matrix by using an activation function layer;

[0252] The third submodule is used to generate initial video frame interaction features according to the video frame query matrix in the video frame feature matrix, the text key value matrix and the text value matrix in the text feature matrix, and the audio key value matrix and the audio value matrix in the audio feature matrix by using an activation function layer;

[0253] A fourth submodule is used to generate initial audio interaction features according to an audio query matrix in the audio feature matrix, a text key value matrix and a text value matrix in the text feature matrix, and a video frame key value matrix and a video frame value matrix in the video frame feature matrix by using an activation function layer;

[0254] The fifth submodule is used to input the initial text interaction features, the initial video frame interaction features and the initial audio interaction features into the bidirectional long short-term memory module for nonlinear mapping, and output the target text interaction features, the target audio interaction features and the target video frame interaction features.

[0255] Furthermore, the second submodule is specifically used for:

[0256] An activation function layer is used to calculate the text query matrix in the text feature matrix and the video frame key value matrix in the video frame feature matrix, and output the text video frame cross-modal attention matrix;

[0257] Performing a matrix multiplication operation on the text-video-frame cross-modal attention matrix and the video frame value matrix in the video frame feature matrix, and outputting a first text interaction feature;

[0258] An activation function layer is used to calculate the text query matrix in the text feature matrix and the audio key value matrix in the audio feature matrix, and output the text-audio cross-modal attention matrix;

[0259] Perform matrix multiplication on the text-audio cross-modal attention matrix and the audio value matrix in the audio feature matrix, and output a second text interaction feature;

[0260] The first text interaction feature and the second text interaction feature are concatenated to generate an initial text interaction feature.

[0261] Furthermore, the third submodule is specifically used for:

[0262] An activation function layer is used to calculate the video frame query matrix in the video frame feature matrix and the text key value matrix in the text feature matrix, and output the video frame text cross-modal attention matrix;

[0263] Perform matrix multiplication on the video frame text cross-modal attention matrix and the text value matrix in the text feature matrix, and output the first video frame interaction feature;

[0264] An activation function layer is used to calculate the video frame query matrix in the video frame feature matrix and the audio key value matrix in the audio feature matrix, and output the video frame audio cross-modal attention matrix;

[0265] Perform matrix multiplication on the audio value matrix in the video frame audio cross-modal attention matrix and the audio feature matrix, and output the second video frame interaction feature;

[0266] The first video frame interaction feature and the second video frame interaction feature are concatenated to generate an initial video interaction feature.

[0267] Furthermore, the fourth submodule is specifically used for:

[0268] An activation function layer is used to calculate the audio query matrix in the audio feature matrix and the text key value matrix in the text feature matrix, and output the audio-text cross-modal attention matrix;

[0269] Perform matrix multiplication on the audio-text cross-modal attention matrix and the text value matrix in the text feature matrix, and output a first audio interaction feature;

[0270] An activation function layer is used to calculate the audio query matrix in the audio feature matrix and the video frame key value matrix in the video frame feature matrix, and output the audio and video frame cross-modal attention matrix;

[0271] Performing a matrix multiplication operation on the audio-video frame cross-modal attention matrix and the video frame value matrix in the video frame feature matrix, and outputting a second audio interaction feature;

[0272] The first audio interaction feature and the second audio interaction feature are concatenated to generate an initial audio interaction feature.

[0273] Further, the representation learning network includes a first feedforward neural layer and a second feedforward neural layer; the multiple modality hidden vectors include multiple first modality hidden vectors and multiple second modality hidden vectors;

[0274] Furthermore, the learning module 504 is specifically used for:

[0275] Inputting the target text interaction features, the target audio interaction features and the target video frame interaction features into the first feedforward neural layer for feature extraction, and outputting a plurality of first modal hidden vectors;

[0276] The target text interaction features, the target audio interaction features and the target video frame interaction features are respectively input into the second feedforward neural layer for feature extraction, and a plurality of second modality hidden vectors are output.

[0277] Furthermore, the feature modality recognition network includes a multi-head self-attention mechanism layer and a prediction layer; a recognition module 505, which is specifically used to:

[0278] Stack multiple modal hidden vectors to generate a modal hidden vector matrix;

[0279] The modal hidden vector matrix is ​​used as the input of the multi-head self-attention mechanism layer and the output is the attention matrix.

[0280] Concatenate multiple hidden vectors in the attention matrix to generate a joint vector;

[0281] The prediction layer is used to predict the joint vector and generate the target multimodal emotion recognition result.

[0282] In an optional embodiment, the device further comprises:

[0283] The first module is used to obtain multimodal emotion data to be trained, input the multimodal emotion data to be trained into the initial multimodal emotion recognition model, and output a plurality of first modal hidden vectors to be trained, a plurality of second modal hidden vectors to be trained, and a multimodal emotion recognition result to be trained;

[0284] The second module is used to determine the model gradient based on the back propagation algorithm according to multiple first modality hidden vectors to be trained, multiple second modality hidden vectors to be trained, and multimodal emotion recognition results to be trained;

[0285] The third module is used to update the initial multimodal emotion recognition model using the model gradient to determine the intermediate multimodal emotion recognition model;

[0286] The fourth module is used to calculate the target loss value according to the multimodal emotion data to be trained by using the intermediate multimodal emotion recognition model and the preset loss function;

[0287] The fifth module is used to determine whether the target loss value converges;

[0288] The sixth module is used to use the intermediate multimodal emotion recognition model as a trained preset multimodal emotion recognition model if convergence occurs.

[0289] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and sub-modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0290] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0291] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0292] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal emotion recognition method, characterized in that: include: Acquire multimodal emotion data to be detected, and input the emotion data to be detected into a preset multimodal emotion recognition model, wherein the preset multimodal emotion recognition model includes a multimodal feature extraction network, a cross-modal feature interaction network, a representation learning network, and a feature modality recognition network; Extracting features of the multimodal emotion data to be detected through the multimodal feature extraction network to generate multimodal emotion perception features; Inputting the multimodal emotion perception features into the cross-modal feature interaction network for feature interaction, and outputting multimodal interaction features; Using the representation learning network to perform feature learning on the multimodal interaction features to generate multiple modal hidden vectors; Emotion recognition is performed on the multiple modal hidden vectors through the characteristic modal recognition network to generate a target multimodal emotion recognition result.

2. The multimodal emotion recognition method according to claim 1, characterized in that: The multimodal emotion data to be detected includes a text sequence to be detected, an audio to be detected, and a video frame image to be detected; the multimodal feature extraction network includes a language representation network, a recursive neural network, and a face detection network; the multimodal emotion perception features include text perception features, audio perception features, and video frame perception features; the multimodal emotion data to be detected is subjected to feature extraction through the multimodal feature extraction network to generate multimodal emotion perception features, including: Extracting text features from the text sequence to be detected through a language representation network to generate text features; Using a recursive neural network to extract audio features from the audio to be detected to generate audio features; Inputting the video frame image to be detected into a face detection network to extract facial features and output a facial action unit sequence; Modeling the temporal features of the facial action unit sequence through a recurrent neural network to generate video frame features; Based on preset position embedding, the text features, the audio features and the video frame features are encoded respectively to generate text-perceptual features, audio-perceptual features and video frame-perceptual features.

3. The multimodal emotion recognition method according to claim 2, characterized in that: The cross-modal feature interaction network includes an activation function layer, a fully connected layer, and a bidirectional long short-term memory module; the multimodal interaction features include target text interaction features, target audio interaction features, and target video frame interaction features; The step of inputting the multimodal emotion perception feature into the cross-modal feature interaction network for feature interaction and outputting the multimodal interaction feature comprises: The text perception feature, the audio perception feature and the video frame perception feature are linearly transformed respectively through a fully connected layer to generate a text feature matrix, an audio feature matrix and a video frame feature matrix; Generate initial text interaction features using an activation function layer according to a text query matrix in the text feature matrix, a video frame key value matrix and a video frame value matrix in the video frame feature matrix, and an audio key value matrix and an audio value matrix in the audio feature matrix; Generate initial video frame interaction features according to a video frame query matrix in the video frame feature matrix, a text key value matrix and a text value matrix in the text feature matrix, and an audio key value matrix and an audio value matrix in the audio feature matrix using an activation function layer; Generate initial audio interaction features using an activation function layer according to an audio query matrix in the audio feature matrix, a text key value matrix and a text value matrix in the text feature matrix, and a video frame key value matrix and a video frame value matrix in the video frame feature matrix; The initial text interaction features, the initial video frame interaction features and the initial audio interaction features are respectively input into a bidirectional long short-term memory module for nonlinear mapping, and target text interaction features, target audio interaction features and target video frame interaction features are output.

4. The multimodal emotion recognition method according to claim 3, characterized in that: The activation function layer is used to generate initial text interaction features according to the text query matrix in the text feature matrix, the video frame key value matrix and the video frame value matrix in the video frame feature matrix, and the audio key value matrix and the audio value matrix in the audio feature matrix, including: An activation function layer is used to calculate a text query matrix in the text feature matrix and a video frame key value matrix in the video frame feature matrix, and output a text-video-frame cross-modal attention matrix; Performing a matrix multiplication operation on the text-video frame cross-modal attention matrix and the video frame value matrix in the video frame feature matrix, and outputting a first text interaction feature; An activation function layer is used to calculate the text query matrix in the text feature matrix and the audio key value matrix in the audio feature matrix, and output a text-audio cross-modal attention matrix; Performing a matrix multiplication operation on the text-audio cross-modal attention matrix and the audio value matrix in the audio feature matrix, and outputting a second text interaction feature; The first text interaction feature and the second text interaction feature are concatenated to generate an initial text interaction feature.

5. The multimodal emotion recognition method according to claim 3, characterized in that: The activation function layer is used to generate initial video frame interaction features according to the video frame query matrix in the video frame feature matrix, the text key value matrix and the text value matrix in the text feature matrix, and the audio key value matrix and the audio value matrix in the audio feature matrix, including: An activation function layer is used to calculate the video frame query matrix in the video frame feature matrix and the text key value matrix in the text feature matrix, and output a video frame text cross-modal attention matrix; Performing a matrix multiplication operation on the video frame text cross-modal attention matrix and the text value matrix in the text feature matrix, and outputting a first video frame interaction feature; An activation function layer is used to calculate the video frame query matrix in the video frame feature matrix and the audio key value matrix in the audio feature matrix, and output a video frame audio cross-modal attention matrix; Performing a matrix multiplication operation on the video frame audio cross-modal attention matrix and the audio value matrix in the audio feature matrix, and outputting a second video frame interaction feature; The first video frame interaction feature and the second video frame interaction feature are spliced ​​to generate an initial video interaction feature.

6. The multimodal emotion recognition method according to claim 3, characterized in that: The activation function layer is used to generate initial audio interaction features according to the audio query matrix in the audio feature matrix, the text key value matrix and the text value matrix in the text feature matrix, and the video frame key value matrix and the video frame value matrix in the video frame feature matrix, including: An activation function layer is used to calculate the audio query matrix in the audio feature matrix and the text key value matrix in the text feature matrix, and output an audio-text cross-modal attention matrix; Performing a matrix multiplication operation on the audio-text cross-modal attention matrix and the text value matrix in the text feature matrix, and outputting a first audio interaction feature; An activation function layer is used to calculate the audio query matrix in the audio feature matrix and the video frame key value matrix in the video frame feature matrix, and output an audio and video frame cross-modal attention matrix; Performing a matrix multiplication operation on the audio-video frame cross-modal attention matrix and the video frame value matrix in the video frame feature matrix, and outputting a second audio interaction feature; The first audio interaction feature and the second audio interaction feature are concatenated to generate an initial audio interaction feature.

7. The multimodal emotion recognition method according to claim 3, characterized in that: The representation learning network includes a first feedforward neural layer and a second feedforward neural layer; the multiple modal hidden vectors include multiple first modal hidden vectors and multiple second modal hidden vectors; the method of using the representation learning network to perform feature learning on the multimodal interaction features to generate multiple modal hidden vectors includes: Inputting the target text interaction feature, the target audio interaction feature and the target video frame interaction feature into a first feedforward neural layer for feature extraction, and outputting a plurality of first modal hidden vectors; The target text interaction features, the target audio interaction features and the target video frame interaction features are respectively input into the second feedforward neural layer for feature extraction, and a plurality of second modality hidden vectors are output.

8. The multimodal emotion recognition method according to claim 1, characterized in that: The characteristic modality recognition network includes a multi-head self-attention mechanism layer and a prediction layer; the emotion recognition is performed on the multiple modality hidden vectors through the characteristic modality recognition network to generate a target multi-modal emotion recognition result, including: Stacking a plurality of the modal hidden vectors to generate a modal hidden vector matrix; Using the modal hidden vector matrix as the input of the multi-head self-attention mechanism layer, and outputting the attention matrix; Concatenate multiple hidden vectors in the attention matrix to generate a joint vector; The prediction layer is used to predict the joint vector to generate a target multimodal emotion recognition result.

9. The multimodal emotion recognition method according to claim 1, characterized in that: The training process of the preset multimodal emotion recognition model includes: Acquire multimodal emotion data to be trained, input the multimodal emotion data to be trained into an initial multimodal emotion recognition model, and output a plurality of first modality hidden vectors to be trained, a plurality of second modality hidden vectors to be trained, and a multimodal emotion recognition result to be trained; Based on the back propagation algorithm, determining the model gradient according to the plurality of first modality hidden vectors to be trained, the plurality of second modality hidden vectors to be trained, and the multimodal emotion recognition results to be trained; Using the model gradient to update the initial multimodal emotion recognition model to determine an intermediate multimodal emotion recognition model; Calculating a target loss value according to the multimodal emotion data to be trained using the intermediate multimodal emotion recognition model and a preset loss function; Determining whether the target loss value converges; If converged, the intermediate multimodal emotion recognition model is used as the trained preset multimodal emotion recognition model.

10. A multimodal emotion recognition device, characterized in that: include: An acquisition module, used for acquiring multimodal emotion data to be detected, and inputting the emotion data to be detected into a preset multimodal emotion recognition model, wherein the preset multimodal emotion recognition model includes a multimodal feature extraction network, a cross-modal feature interaction network, a representation learning network, and a feature modality recognition network; A generating module, used for performing feature extraction on the multimodal emotion data to be detected through the multimodal feature extraction network to generate multimodal emotion perception features; An output module, used for inputting the multimodal emotion perception features into the cross-modal feature interaction network for feature interaction, and outputting multimodal interaction features; A learning module, used to perform feature learning on the multimodal interaction features using the representation learning network to generate multiple modal hidden vectors; The recognition module is used to perform emotion recognition on the multiple modal hidden vectors through the characteristic modal recognition network to generate a target multimodal emotion recognition result.

Images