Multi-modal session emotion recognition method and system based on missing modal reconstruction

Abstract

The invention provides a multi-modal session emotion recognition method and system based on missing modal reconstruction. The method comprises the following steps: collecting multi-modal emotion data; the multi-mode emotion data is preprocessed; reconstructing missing modal information for the preprocessed data by using a label coding and feature fusion method; projecting the reconstructed features of the audio mode and the video mode to a text mode space, and training an emotion recognition model; and inputting the reconstructed three modal features into an emotion recognition model for emotion classification. The invention provides a modal reconstruction and text space projection method, and the performance of an emotion recognition model under the condition of modal deficiency is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a multimodal conversation emotion recognition method and system based on missing modality reconstruction. Background Art

[0002] In recent years, with the rapid development of deep learning technology, emotion recognition has become a hot topic, and it has broad application prospects in various fields such as education, services, finance, and daily life. Generally speaking, emotion analysis technology includes two categories: unimodal analysis technology and multimodal analysis technology. Unimodal analysis technology only relies on information from a single modality, such as voice information, text, facial expressions, etc. However, models that only use a single modality often perform poorly. Therefore, although unimodal emotion analysis technology has developed relatively maturely, human emotional communication is essentially multimodal. Multimodal emotion analysis technology uses data information from modalities such as text, voice, and images, and effectively integrates and analyzes the information of these modalities through machine learning methods. Compared with emotion analysis technology that only obtains data from a single modality from a single information source, multimodal emotion analysis can more accurately identify and understand human emotional states.

[0003] Most existing multimodal sentiment analysis models usually use the Transformer architecture to extract data features of different modalities, and the Long Short Term Memory (LSTM) or Graph Convolutional Network (GCN) to capture temporal relationships and perform modal fusion. However, these methods can only use partial modal information for training when the modality is missing, and cannot supplement the missing modality, resulting in a significant decrease in model performance when the modality is missing.

[0004] Patent document CN117828078A discloses a multimodal conversation emotion recognition method based on a two-layer relationship graph, which uses a two-layer relationship graph, a global affinity graph constructed based on node similarity, and a local relationship graph constructed using context information. Compared with this method, our method uses the technical route of modal reconstruction and text space projection to achieve emotion recognition in any modal combination, which is significantly different from the patent in terms of technical route.

[0005] The paper Multimodal Reconstruct and Align Net for Missing Modality Problem in Sentiment Analysis. (In MultiMedia Modeling: 29th International Conference, MMM 2023) points out that the missing key modal information leads to inconsistent sentiment recognition, and proposes a method for missing modal projection. In terms of supplementing the missing modality, compared with the paper, our technical route uses modal encoding, modal information fusion, and text space projection to supplement the missing modality, which is significantly different from the method in the reconstruction of the missing modality.

[0006] In view of the shortcomings of previous work and methods, this paper proposes a new multimodal conversation emotion recognition model based on missing modality reconstruction. This method uses a label encoding method to prompt a specific modality combination; at the same time, the inputs of the three modalities (the missing modality is zero input) are spliced ​​and passed through a layer of network, so that the missing modality can sense the information of other modalities; in addition, the data of audio and video modalities are projected into the text space so that they have similar representations, thereby improving the performance when the text modality is missing. Summary of the invention

[0007] In view of the defects in the prior art, the object of the present invention is to provide a multimodal conversation emotion recognition method and system based on missing modality reconstruction.

[0008] A multimodal conversation emotion recognition method based on missing modality reconstruction provided by the present invention includes:

[0009] Step S1: Collect multimodal emotion data;

[0010] Step S2: preprocessing the multimodal emotion data;

[0011] Step S3: Reconstruct the missing modal information using label encoding and feature fusion methods on the preprocessed data;

[0012] Step S4: projecting the reconstructed features of the audio modality and the video modality into the text modality space to train the emotion recognition model;

[0013] Step S5: Input the reconstructed three modal features into the emotion recognition model for emotion classification.

[0014] Preferably, the multimodal emotion data includes video, audio and text information in the conversation.

[0015] Preferably, step S2 includes using OpenSmile to process the audio data to obtain audio features, using OpenFace to process the video data to obtain video features, and using sBERT to process the text data to obtain text features. If any modality is missing, it is processed as a zero vector.

[0016] Preferably, the step S3 comprises:

[0017] Step S3.1: Label the modal information to record the current modal combination. Pass the label through a fully connected network to convert the 3-bit modal information label into a vector with the same length as the modal feature, and obtain the embedding of the modal information. The formula is as follows:

[0018] x m =FC([I a ,I t ,I v ])

[0019] Among them, I s ∈{0,1},s∈{a,t,v} represents the label encoding of audio a, text t, and video v modalities;

[0020] Step S3.2: Align and concatenate the features of the three modalities, and pass through a fully connected network with two neurons to transform the vector of the combined three modal features into a vector with the same length as the modal feature to obtain the embedding of the modal feature. The formula is as follows:

[0021] x c =FC(Concat([x a ,x t ,x v ]))

[0022] Among them, x τ ,τ∈{a,t,v} represents the feature vectors of audio, text and video modalities respectively;

[0023] Step S3.3: After the modal information embedding, modal feature embedding and original feature embedding are added together, they are passed through a fully connected network with two neurons to obtain the reconstructed modal feature. The formula is as follows:

[0024] f τ =x τ +x m +x c ,τ∈{a,t,v}

[0025] The distance between the reconstructed features and the features obtained by the pre-trained model is calculated and used as the loss function for supervised learning. The loss function formula is as follows:

[0026]

[0027] Among them, f τ ,τ∈{a,t,v} represents the feature vector obtained by the reconstruction module, Represents the feature vector obtained from the complete modality information in the pre-trained baseline model.

[0028] Preferably, step S4 includes projecting the video features and the audio features into the text space through a layer of fully connected network, which can be expressed by the following formula:

[0029]

[0030] in, represents the trainable parameters in the fully connected neural network, f τ is a feature located in the text vector space;

[0031] The distance between the projected features and the text features obtained by the pre-trained model is calculated and used as the loss function for supervised learning. The loss function is expressed as follows:

[0032]

[0033] in, represents the audio and video feature vectors projected into the text vector space, Represents the text feature vector in the pre-trained baseline model.

[0034] Preferably, the fully connected network contains two neurons.

[0035] A multimodal conversation emotion recognition system based on missing modality reconstruction provided by the present invention includes:

[0036] Module M1: Collecting multimodal emotion data;

[0037] Module M2: preprocessing the multimodal emotion data;

[0038] Module M3: Reconstruct missing modal information using label encoding and feature fusion systems on preprocessed data;

[0039] Module M4: Project the features of the reconstructed audio and video modalities into the text modality space to train the emotion recognition model;

[0040] Module M5: Input the reconstructed three modal features into the emotion recognition model for emotion classification.

[0041] Preferably, the module M3 comprises:

[0042] Module M3.1: Label the modal information to record the current modal combination. Pass the label through a fully connected network to convert the 3-bit modal information label into a vector with the same length as the modal feature, and obtain the embedding of the modal information. The formula is as follows:

[0043] x m =FC([I a ,I t ,I v ])

[0044] Among them, I s ∈{0,1},s∈{a,t,v} represents the label encoding of audio a, text t, and video v modalities;

[0045] Module M3.2: Align and concatenate the features of the three modalities, and pass a fully connected network with two neurons to transform the vector of the combined three modal features into a vector with the same length as the modal feature to obtain the embedding of the modal feature. The formula is as follows:

[0046] x c =FC(Concat([x a ,x t ,x v ]))

[0047] Among them, x τ ,τ∈{a,t,v} represents the feature vectors of audio, text and video modalities respectively;

[0048] Module M3.3: After the modal information embedding, modal feature embedding and original feature embedding are added together, they are passed through a fully connected network with two neurons to obtain the reconstructed modal features. The formula is as follows:

[0049] f τ =x τ +x m +x c ,τ∈{a,t,v}

[0050] The distance between the reconstructed features and the features obtained by the pre-trained model is calculated and used as the loss function for supervised learning. The loss function formula is as follows:

[0051]

[0052] Among them, f τ ,τ∈{a,t,v} represents the feature vector obtained by the reconstruction module, Represents the feature vector obtained from the complete modality information in the pre-trained baseline model.

[0053] Preferably, the module M4 includes projecting the video features and the audio features into the text space through a layer of fully connected network, which can be expressed as follows:

[0054]

[0055] in, represents the trainable parameters in the fully connected neural network, f τ is a feature located in the text vector space;

[0056] The distance between the projected features and the text features obtained by the pre-trained model is calculated and used as the loss function for supervised learning. The loss function is expressed as follows:

[0057]

[0058] in, represents the audio and video feature vectors projected into the text vector space, Represents the text feature vector in the pre-trained baseline model.

[0059] Preferably, the fully connected network contains two neurons.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] 1. The present invention uses a modal reconstruction method to enable the multimodal sentiment analysis model to recognize information of any modal combination, while the traditional multimodal model cannot perform recognition in the absence of model information.

[0062] 2. The present invention uses a text space projection method to significantly improve the robustness of the model when the text modality is missing. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0064] Figure 1 It is a schematic diagram of the working method of the present invention;

[0065] Figure 2 It is a module framework diagram of the present invention. DETAILED DESCRIPTION

[0066] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0067] Example 1

[0068] According to a multimodal conversation emotion recognition method based on missing modality reconstruction provided by the present invention, modal features are reconstructed by label encoding and modal information fusion, and video and audio modal information are projected into text modal space to complete the missing modal information supplement. Figure 1 As shown, including:

[0069] Step S1: Collect multimodal emotion data. The multimodal emotion data includes video, audio and text information in the conversation. Step S1 includes eliciting the emotional response of the subject in the conversation and recording the data in three modes: video, audio and text transcription.

[0070] Step S2: Preprocess the multimodal emotional data. Step S2 includes using OpenSmile to process the audio data to obtain audio features, using OpenFace to process the video data to obtain video features, and using sBERT to process the text data to obtain text features. If any modality is missing, it is processed as a zero vector. In other words, any missing modality is represented by a zero vector as a feature. Specifically, OpenSmile is used to process the audio data to obtain 100-dimensional audio features, OpenFace is used to process the video data to obtain 512-dimensional facial image features, and sBERT is used to process the text data to obtain 768-dimensional text features. If any modality is missing, it is processed as a zero vector. The data set is divided into 7 groups according to the modality combination, of which 3 groups are bimodal missing, 3 groups are unimodal missing, and 1 group is full modality, and is divided into training sets and test sets in proportion.

[0071] Step S3: Reconstruct the missing modal information using label encoding and feature fusion methods on the preprocessed data.

[0072] The step S3 comprises:

[0073] Step S3.1: Modal encoding, label the modal information to record the current modal combination, pass the label through a layer of fully connected network containing two neurons, convert the 3-bit modal information label into a vector with the same length as the modal feature, and obtain the embedding of the modal information. Specifically, if the current modality exists, it is recorded as "1", and if it does not exist, it is recorded as "0", and a 3-bit label is obtained. The label is passed through a layer of fully connected network to obtain the embedding of the modal information with the same embedding dimension as the original modal feature, which is expressed as follows:

[0074] x m =FC([I a ,I t ,Iv ])

[0075] Among them, I s ∈{0,1},s∈{a,t,v} represents the label encoding of audio (a), text (t), and video (v) modalities.

[0076] Step S3.2: Modal information fusion, align the features of the three modalities and concatenate them, and pass through a layer of fully connected network containing two neurons to convert the vector of the three modal features into a vector with the same length as the modal feature to obtain the embedding of the modal feature. At the same time, align the features of the three modalities and concatenate them, and pass through a layer of fully connected network to obtain the embedding of the modal feature fusion with the same embedding dimension as the original modal feature, which is expressed by the following formula:

[0077] x c =FC(Concat([x a ,x t ,x v ]))

[0078] Among them, x τ ,τ∈{a,t,v} represents the feature vectors of audio, text and video modalities respectively.

[0079] Step S3.3: After the modal information embedding, modal feature embedding and original feature embedding are added, they are passed through a layer of fully connected network containing two neurons to obtain the reconstructed modal features, and the distance between the reconstructed features and the features obtained by the pre-trained model is calculated, which is used as the loss function for supervised learning. After the modal information embedding, modal feature fusion embedding and original feature embedding are spliced, they are passed through a layer of fully connected network to obtain the reconstructed modal features, which can be expressed as follows:

[0080] f τ =x τ +x m +x c ,τ∈{a,t,v}

[0081] The distance between the reconstructed features and the features obtained by the pre-trained model is calculated and used as the loss function for supervised learning. The loss function is expressed as follows:

[0082]

[0083] Among them, f τ ,τ∈{a,t,v} represents the feature vector obtained by the reconstruction module, Represents the feature vector obtained from the complete modality information in the pre-trained baseline model.

[0084] Step S4: Project the reconstructed audio and video features to the text modality space to train the emotion recognition model. Step S4 includes text space projection, projecting the video features and audio features to the text space through a layer of fully connected network containing two neurons, converting the video and audio features into vectors with the same length as the text features, and calculating the distance between the projected features and the text features obtained by the pre-trained model, which is used as the loss function for supervised learning. The video features and audio features are projected to the text space through a layer of fully connected network containing two neurons, which is expressed by the following formula:

[0085]

[0086] in, represents the trainable parameters in the fully connected neural network, f τ is a feature in the text vector space

[0087] The distance between the projected features and the text features obtained by the pre-trained model is calculated and used as the loss function for supervised learning. The loss function is expressed as follows:

[0088]

[0089] in, represents the audio and video feature vectors projected into the text vector space, Represents the text feature vector in the pre-trained baseline model.

[0090] Step S5: Input the reconstructed three modal features into the emotion recognition model for emotion classification. The step S5 includes inputting the processed three modal features into the pre-trained emotion classification model to obtain a classification result.

[0091] The purpose of the present invention is to address the problem that the performance of the prior art is significantly degraded when the modality is missing. The present invention proposes a new modality reconstruction method, including modality encoding, modality information fusion, and text space projection, which can fill in the missing modal information and enable the model to recognize information of any modal combination.

[0092] Example 2

[0093] The present invention also provides a multimodal conversation emotion recognition system based on missing modality reconstruction. The multimodal conversation emotion recognition system based on missing modality reconstruction can be implemented by executing the process steps of the multimodal conversation emotion recognition method based on missing modality reconstruction, that is, those skilled in the art can understand the multimodal conversation emotion recognition method based on missing modality reconstruction as a preferred implementation of the multimodal conversation emotion recognition system based on missing modality reconstruction.

[0094] A multimodal conversation emotion recognition method based on missing modality reconstruction provided by the present invention includes:

[0095] Module M1: Collecting multimodal emotion data. The multimodal emotion data includes video, audio and text information in the conversation.

[0096] Module M2: Preprocessing the multimodal emotion data. Module M2 includes using OpenSmile to process the audio data to obtain audio features, using OpenFace to process the video data to obtain video features, and using sBERT to process the text data to obtain text features. If any modality is missing, it is processed as a zero vector.

[0097] Module M3: Reconstruct the missing modal information using label encoding and feature fusion methods for the preprocessed data. The module M3 includes: Module M3.1: Label the modal information to record the current modal combination, pass the label through a layer of fully connected network, convert the 3-bit modal information label into a vector with the same length as the modal feature, and obtain the embedding of the modal information. The formula is as follows:

[0098] x m =FC([I a ,I t ,I v ])

[0099] Among them, I s ∈{0,1},s∈{a,t,v} represents the label encoding of the audio a, text t, and video v modalities. Module M3.2: Align and concatenate the features of the three modalities, and transform the vector of the combined features of the three modalities into a vector of the same length as the modal feature through a fully connected network with two neurons to obtain the embedding of the modal feature. The formula is as follows:

[0100] x c =FC(Concat([x a ,x t ,x v ]))

[0101] Among them, x τ ,τ∈{a,t,v} represents the feature vectors of audio, text and video modalities respectively. Module M3.3: After adding the modal information embedding, modal feature embedding and original feature embedding, a fully connected network with two neurons is passed through to obtain the reconstructed modal features. The formula is as follows:

[0102] f τ =x τ +x m +x c ,τ∈{a,t,v}

[0103] The distance between the reconstructed features and the features obtained by the pre-trained model is calculated and used as the loss function for supervised learning. The loss function formula is as follows:

[0104]

[0105] Among them, f τ ,τ∈{a,t,v} represents the feature vector obtained by the reconstruction module, Represents the feature vector obtained from the complete modality information in the pre-trained baseline model.

[0106] Module M4: Project the reconstructed audio and video features into the text modality space to train the emotion recognition model. The module M4 includes projecting the video features and audio features into the text space through a fully connected network, which can be expressed as follows:

[0107]

[0108] in, represents the trainable parameters in the fully connected neural network, f τ is a feature located in the text vector space. The distance between the projected feature and the text feature obtained by the pre-training model is calculated and used as the loss function for supervised learning. The loss function is expressed as follows:

[0109]

[0110] in, represents the audio and video feature vectors projected into the text vector space, Represents the text feature vector in the pre-trained baseline model. The fully connected network contains two neurons.

[0111] Module M5: Input the reconstructed three modal features into the emotion recognition model for emotion classification.

[0112] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, the system and its various devices, modules, and units provided by the present invention can be completely realized in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps to realize the same functions. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0113] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A multimodal conversation emotion recognition method based on missing modality reconstruction, characterized in that: include: Step S1: Collect multimodal emotion data; Step S2: preprocessing the multimodal emotion data; Step S3: Reconstruct the missing modal information using label encoding and feature fusion methods on the preprocessed data; Step S4: Projecting the reconstructed features of the audio modality and the video modality into the text modality space to train the emotion recognition model; Step S5: Input the reconstructed three modal features into the emotion recognition model for emotion classification.

2. The multimodal conversation emotion recognition method based on missing modality reconstruction according to claim 1 is characterized in that: The multimodal emotion data includes video, audio and text information in the conversation.

3. The multimodal conversation emotion recognition method based on missing modality reconstruction according to claim 1 is characterized in that: The step S2 includes using OpenSmile to process the audio data to obtain audio features, using OpenFace to process the video data to obtain video features, and using sBERT to process the text data to obtain text features. If any modality is missing, it is processed as a zero vector.

4. The multimodal conversation emotion recognition method based on missing modality reconstruction according to claim 1 is characterized in that: The step S3 comprises: Step S3.1: Label the modal information to record the current modal combination. Pass the label through a fully connected network to convert the 3-bit modal information label into a vector with the same length as the modal feature, and obtain the embedding of the modal information. The formula is as follows: x m =FC([I a ,I t ,I v ]) Among them, I s ∈{0,1},s∈{a,t,v} represents the label encoding of audio a, text t, and video v modalities; Step S3.2: Align and concatenate the features of the three modalities, and pass through a fully connected network with two neurons to transform the vector of the combined three modal features into a vector with the same length as the modal feature to obtain the embedding of the modal feature. The formula is as follows: x c =FC(Concat([x a ,x t ,x v ])) Among them, x τ ,τ∈{a,t,v} represents the feature vectors of audio, text and video modalities respectively; Step S3.3: After the modal information embedding, modal feature embedding and original feature embedding are added together, they are passed through a fully connected network with two neurons to obtain the reconstructed modal feature. The formula is as follows: f τ =x τ +x m +x c ,τ∈{a,t,v} The distance between the reconstructed features and the features obtained by the pre-trained model is calculated and used as the loss function for supervised learning. The loss function formula is as follows: Among them, f τ ,τ∈{a,t,v} represents the feature vector obtained by the reconstruction module, Represents the feature vector obtained from the complete modality information in the pre-trained baseline model.

5. The multimodal conversation emotion recognition method based on missing modality reconstruction according to claim 1 is characterized in that: The step S4 includes projecting the video features and the audio features into the text space through a fully connected network layer, which is expressed by the following formula: in, represents the trainable parameters in the fully connected neural network, f τ is a feature located in the text vector space; The distance between the projected features and the text features obtained by the pre-trained model is calculated and used as the loss function for supervised learning. The loss function is expressed as follows: in, represents the audio and video feature vectors projected into the text vector space, Represents the text feature vector in the pre-trained baseline model.

6. The multimodal conversation emotion recognition method based on missing modality reconstruction according to claim 4 or 5, characterized in that: The fully connected network contains two neurons.

7. A multimodal conversation emotion recognition system based on missing modality reconstruction, characterized in that: include: Module M1: Collecting multimodal emotion data; Module M2: preprocessing the multimodal emotion data; Module M3: Reconstruct missing modal information using label encoding and feature fusion systems on preprocessed data; Module M4: Project the features of the reconstructed audio and video modalities into the text modality space to train the emotion recognition model; Module M5: Input the reconstructed three modal features into the emotion recognition model for emotion classification.

8. The multimodal conversation emotion recognition system based on missing modality reconstruction according to claim 7 is characterized in that: The module M3 comprises: Module M3.1: Label the modal information to record the current modal combination. Pass the label through a fully connected network to convert the 3-bit modal information label into a vector with the same length as the modal feature, and obtain the embedding of the modal information. The formula is as follows: x m =FC([I a ,I t ,I v ]) Among them, I s ∈{0,1},s∈{a,t,v} represents the label encoding of audio a, text t, and video v modalities; Module M3.2: Align and concatenate the features of the three modalities, and pass a fully connected network with two neurons to transform the vector of the combined three modal features into a vector with the same length as the modal feature to obtain the embedding of the modal feature. The formula is as follows: x c =FC(Concat([x a ,x t ,x v ])) Among them, x τ ,τ∈{a,t,v} represents the feature vectors of audio, text and video modalities respectively; Module M3.3: After the modal information embedding, modal feature embedding and original feature embedding are added together, they are passed through a fully connected network with two neurons to obtain the reconstructed modal features. The formula is as follows: f τ =x τ +x m +x c ,τ∈{a,t,v} The distance between the reconstructed features and the features obtained by the pre-trained model is calculated and used as the loss function for supervised learning. The loss function formula is as follows: Among them, f τ ,τ∈{a,t,v} represents the feature vector obtained by the reconstruction module, Represents the feature vector obtained from the complete modality information in the pre-trained baseline model.

9. The multimodal conversation emotion recognition system based on missing modality reconstruction according to claim 7, characterized in that: The module M4 includes projecting the video features and audio features into the text space through a fully connected network, which can be expressed as follows: in, represents the trainable parameters in the fully connected neural network, f τ is a feature located in the text vector space; The distance between the projected features and the text features obtained by the pre-trained model is calculated and used as the loss function for supervised learning. The loss function is expressed as follows: in, represents the audio and video feature vectors projected into the text vector space, Represents the text feature vector in the pre-trained baseline model.

10. The multimodal conversation emotion recognition system based on missing modality reconstruction according to claim 8 or 9, characterized in that: The fully connected network contains two neurons.

Images