Multi-modal emotion recognition method and system

Abstract

The invention discloses a multi-modal emotion recognition method and system, and relates to the technical field of image processing, and the method comprises the following steps: obtaining multi-modal emotion data and missing emotion data, and carrying out the labeling, and obtaining the labeled multi-modal emotion data and missing emotion data, the multi-modal emotion data comprises image, voice and text features; inputting the labeled multi-modal emotion data and missing emotion data into a pre-established trained deep neural network model based on a distillation mechanism and comparative learning, and outputting to obtain fusion features and updated single-modal features; and generating positive and negative samples based on the fusion features and the updated single-mode features, and inputting the fusion features to a full connection layer based on the positive and negative samples to obtain an emotion recognition prediction result.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multimodal emotion recognition method and system. Background Art

[0002] Multimodal sentiment analysis has always been a hot research topic in sentiment computing. It has become a trend for people to express their emotions and opinions through videos on social media. Researchers use the massive amount of video data generated every day on social media to carry out a lot of research work to achieve intelligent human-computer interaction, driver fatigue detection, and family elderly care companionship. Multimodal sentiment analysis extracts rich emotional information from visual, voice, and text data, and can more effectively judge the emotional state of the object than a single-modal system. This is because different modal data complement each other and can effectively eliminate the shortcomings of ambiguity in the emotions expressed by a single modality. However, in practical applications, modal data is usually incomplete, for example, the surrounding environmental noise interferes with the speaker's voice, and the image taken by the camera is blurred due to sensor problems. These inevitable factors greatly affect the accuracy of multimodal sentiment recognition.

[0003] Most existing studies assume that one modality is missing, ignoring the more common situation of multiple modalities missing and uncertain. Such methods usually adopt a generation-based approach, which requires training a new model each time to adapt to the situation of each missing modality, which is obviously not flexible enough. In addition, some studies have adopted a joint representation approach. Although it solves the problem of multiple modal missing, most studies fix the missing samples during the training process, thus limiting the ability to handle more complex situations. Summary of the invention

[0004] In order to solve the deficiencies mentioned in the above background technology, the purpose of the present invention is to provide a multimodal emotion recognition method and system, which can solve the problem of random feature missing and improve the prediction performance of the network model.

[0005] In a first aspect, the purpose of the present invention can be achieved by the following technical solution: a multimodal emotion recognition method, the method comprising the following steps:

[0006] Acquire multimodal emotion data, annotate them, and randomly mask the annotated multimodal emotion data to obtain missing emotion data, wherein the multimodal emotion data includes image, voice, and text features;

[0007] The annotated multimodal sentiment data and missing sentiment data are input into a pre-established trainable deep neural network model based on distillation mechanism and contrastive learning, and the fused features and updated unimodal features are output;

[0008] Based on the fused features and the updated unimodal features, positive and negative samples are generated. Based on the positive and negative samples, the fused features are input into the fully connected layer to obtain the emotion recognition prediction results.

[0009] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: the pre-established trainable deep neural network model based on distillation mechanism and contrastive learning includes a teacher network, a student network, a distillation algorithm module and a contrastive prediction coding module.

[0010] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the construction process of the pre-established trainable deep neural network model based on distillation mechanism and contrastive learning includes the following steps:

[0011] The complete modality data is used as the input of the teacher network model for training, and the teacher fusion features containing comprehensive multimodal information are obtained, and the pre-training weights are saved;

[0012] The missing features are used as the input of the student model for training to obtain the student fusion features;

[0013] Input the teacher fusion features and the student fusion features into the distillation algorithm together;

[0014] The missing data is then input into the contrastive prediction coding module through the unimodal features of the encoder of the student model and the student fusion features, retaining valuable key task information, and finally generating a trainable deep neural network model based on distillation mechanism and contrastive learning.

[0015] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: the teacher network model includes a feature extraction module, a feature reconstruction module and a late fusion module; wherein the feature extraction module is a feature extraction network based on mutual attention and self-attention, and is stacked by 3 one-dimensional convolutional layers, 3 position embedding layers, 6 cross-modal attention encoders and 6 self-attention encoders and 3 splicing operations, the feature reconstruction module is used to reconstruct the original complete sequence from the sequence generated by the feature extraction module, and the late fusion module obtains the teacher fusion result, including 3 bidirectional gated recurrent units, 3 activation functions, 3 one-dimensional convolutional networks, 3 maximum pooling and 1 splicing operation.

[0016] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: the student model is composed of a unimodal encoder and a global-local cross-modal fusion module, wherein the unimodal encoder extracts modal discourse representation and time step features, and inputs the features into a linear layer to map them into the same dimension and concatenates the discourse-level representations of the three modalities to obtain a global multimodal representation; the global-local cross-modal fusion module gradually improves the information interaction between each layer in a multi-layer stacking manner, and the time step features from each modality and the global modal representation are subjected to cross-modal attention, self-attention operations and feedforward neural networks; after the multi-layer cross-modal attention modules are stacked, the time dimension of the final global multimodal context shared representation is flattened, and then concatenated with the extracted unimodal representation, mapped to a low-dimensional space through a linear layer to obtain a student fusion result, and finally the fused representation is input into a multi-layer perceptron to output a prediction result.

[0017] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: the distillation algorithm embeds comprehensive multimodal knowledge into the student model to utilize complete modal information to assist learning with missing data.

[0018] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: the comparative prediction coding module uses a scoring function acting on two normalized vectors to measure the correlation between the unimodal features and the fusion representation, and the unimodal-fusion representations in the same fusion relationship are regarded as positive sample pairs, and other unimodal and fusion representations in the same batch are regarded as negative sample pairs, and the comparative losses of the last three modalities and the fusion representation are added to form the final comparative loss result.

[0019] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: inputting the modal data that is lost in proportion into the model, and outputting the probability value of the emotion category. If the emotion polarity result corresponding to the maximum probability is the same as the actual emotion category, it means that the prediction is correct, and the classification accuracy of the multimodal emotion recognition is obtained, that is, the proportion of correctly predicted samples to the total samples.

[0020] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the pre-established trainable deep neural network model based on distillation mechanism and contrastive learning is expressed as follows:

[0021] KC4RM=T-Net+S-Net+KDM+CLM

[0022] KC4RM is the general name of the model, T-Net is the teacher network, S-Net is the student network, KDM is the knowledge distillation algorithm module, and CLM is contrastive predictive coding.

[0023] In a second aspect, in order to achieve the above-mentioned object, the present invention discloses a multimodal emotion recognition system, comprising:

[0024] A data processing module, used to obtain multimodal emotion data, annotate it, and randomly mask the annotated multimodal emotion data to obtain missing emotion data, wherein the multimodal emotion data includes image, voice and text features;

[0025] The feature extraction module is used to input the annotated multimodal sentiment data and missing sentiment data into a pre-established trainable deep neural network model based on distillation mechanism and contrastive learning, and output fused features and updated unimodal features;

[0026] The recognition prediction module is used to generate positive and negative samples based on the fused features and the updated unimodal features, and input the fused features into the fully connected layer based on the positive and negative samples to obtain the emotion recognition prediction results.

[0027] Beneficial effects of the present invention:

[0028] The present invention effectively utilizes the characteristics of the knowledge distillation transfer mechanism, and uses the comprehensive modal information transferred by the teacher network to assist the training of the student network for learning with missing data. Positive and negative sample pairs are constructed between the fusion features and unimodal features of the student network, and valuable information related to the task is retained to better enhance the correlation between the fusion features and the unimodal features. In summary, the distillation mechanism and contrastive learning network proposed in the present invention improve the accuracy of emotion recognition in real scenarios with missing data, which is conducive to the future development of the field of emotional computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 It is a schematic flow chart of the method of the present invention;

[0031] Figure 2 It is a schematic diagram of the workflow of the present invention;

[0032] Figure 3 It is a schematic diagram of the system structure of the present invention;

[0033] Figure 4 It is an experimental effect diagram of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] Embodiment 1:

[0036] The following is an introduction to the relevant terms involved in the embodiments of the present application:

[0037] Multimodal features: Multimodal features refer to features extracted from multiple heterogeneous modal data. These features can come from different data sources, such as images, text, speech, etc. The importance of multimodal features lies in their ability to provide more comprehensive and rich information, which helps improve the accuracy of recognition and classification. For example, in biometrics, multimodal technology can improve the accuracy and security of authentication by fusing multiple biometric features such as fingerprints, faces, and irises.

[0038] like Figure 1 As shown, a multimodal emotion recognition method comprises the following steps:

[0039] S101: Acquire multimodal emotion data, annotate it, and randomly mask the annotated multimodal emotion data to obtain missing emotion data, wherein the multimodal emotion data includes image, voice and text features;

[0040] The original modal data needs to be preprocessed, in which the text modality is extracted by the pre-trained Bert model into a feature dimension of 768. For foreign data sets, COVAREP is used to extract various low-level audio features for audio features, and the facial features are extracted by the tool Facet. The Chinese data set uses Librosa to extract audio features and OpenFace to extract facial units. In order to obtain the random feature missing in the real scene, the masking function is used to randomly mask the complete sequence, and the masked features are set to 0.

[0041] S102: inputting the annotated multimodal sentiment data and the missing sentiment data into a pre-established trainable deep neural network model based on a distillation mechanism and contrastive learning, and outputting fused features and updated unimodal features;

[0042] The pre-established trainable deep neural network model based on distillation mechanism and contrastive learning includes a teacher network, a student network, a distillation algorithm module and a contrastive predictive coding module.

[0043] The construction process of the pre-built trainable deep neural network model based on distillation mechanism and contrastive learning includes the following steps:

[0044] The complete modality data is used as the input of the teacher network model for training, and the teacher fusion features containing comprehensive multimodal information are obtained, and the pre-training weights are saved;

[0045] The missing features are used as the input of the student model for training to obtain the student fusion features;

[0046] Input the teacher fusion features and the student fusion features into the distillation algorithm together;

[0047] The missing data is then input into the contrastive prediction coding module through the unimodal features of the encoder of the student model and the student fusion features, retaining valuable key task information, and finally generating a trainable deep neural network model based on distillation mechanism and contrastive learning.

[0048] The teacher network model includes a feature extraction module, a feature reconstruction module and a late fusion module; the feature extraction module is a feature extraction network based on mutual attention and self-attention, which is stacked by 3 one-dimensional convolutional layers, 3 position embedding layers, 6 cross-modal attention encoders and 6 self-attention encoders and 3 splicing operations. The feature reconstruction module is used to reconstruct the original complete sequence from the sequence generated by the feature extraction module. The late fusion module obtains the teacher fusion result, including 3 bidirectional gated recurrent units, 3 activation functions, 3 one-dimensional convolutional networks, 3 maximum pooling and 1 splicing operation.

[0049] The student model consists of a unimodal encoder and a global-local cross-modal fusion module, in which the unimodal encoder extracts modal discourse representation and time step features, and inputs the features into a linear layer to map them into the same dimension and concatenates the discourse-level representations of the three modalities to obtain a global multimodal representation; the global-local cross-modal fusion module uses a multi-layer stacking approach to gradually improve the information interaction between each layer, and the time step features from each modality and the global modal representation are subjected to cross-modal attention, self-attention operations and feedforward neural networks; after stacking multiple layers of cross-modal attention modules, the time dimension of the final global multimodal context shared representation is flattened, and then concatenated with the extracted unimodal representation, mapped to a low-dimensional space through a linear layer to obtain the student fusion result, and finally the fused representation is input into a multi-layer perceptron to output the prediction result.

[0050] The distillation algorithm embeds comprehensive multimodal knowledge into the student model to utilize complete modality information to assist learning with missing data.

[0051] The contrastive predictive coding module uses a scoring function acting on two normalized vectors to measure the correlation between the unimodal features and the fusion representation. The unimodal-fusion representations in the same fusion relationship are regarded as positive sample pairs, and other unimodal and fusion representations in the same batch are regarded as negative sample pairs. Finally, the contrast losses of the last three modalities and the fusion representation are added together to form the final contrast loss result.

[0052] The modal data with loss in proportion is input into the model, and the output is the probability value of the emotion category. If the emotion polarity result corresponding to the maximum probability is the same as the true emotion category, it means that the prediction is correct, and the classification accuracy of the multimodal emotion recognition is obtained, that is, the proportion of samples with correct prediction to the total samples.

[0053] The pre-established trainable deep neural network model based on distillation mechanism and contrastive learning is expressed as follows:

[0054] KC4RM=T-Net+S-Net+KDM+CLM

[0055] KC4RM is the general name of the model, T-Net is the teacher network, S-Net is the student network, KDM is the knowledge distillation algorithm module, and CLM is the comparative prediction coding. Figure 2 As shown in Figure 1, the deep neural network model consists of a teacher network, a student network, knowledge distillation and contrastive prediction coding modules. The teacher model pre-uses the complete modal data. Training, where T m is the sequence length, d m is the feature dimension of the modality m∈{t,v,a}. The overall model in the second stage takes the missing modality as input. The teacher network encodes the modalities of different sequence lengths into the same dimension, obtains an updated single-modal representation, and obtains the result by reconstruction and post-fusion with the original features. The distillation algorithm module is based on relational consistency to transfer teacher knowledge, and its input is the teacher fusion F t - The fusion layer of the student represents F s The student network focuses on the deep interaction between missing modalities, that is, using the discourse level of each modality as the context of global multimodality and performing cross-modal attention with local unimodal features. At the same time, the contrastive learning strategy is introduced in the student network to maximize the mutual information between the fusion result and the input unimodality, that is, to minimize the contrastive learning loss function.

[0056] The first stage of training is to use the complete modality features as the input of the teacher network, and finally obtain the teacher's fusion results through the late fusion part. The second stage of training is to use the missing features as the input of the student network. During the training process, the distillation loss function is used to assist the missing modality in learning the semantics of the complete modality. In addition, the contrastive learning loss function is added to deal with the problem of information redundancy from the input to the fusion layer.

[0057] S103: Based on the fused features and the updated unimodal features, positive and negative samples are generated, and the fused features are input into the fully connected layer based on the positive and negative samples to obtain the emotion recognition prediction results.

[0058] During the training of the student model, task-related information is injected into the fusion result. The goal is to make the fused representation retain the most important part of each single modality as much as possible, and use a scoring function acting on two normalized vectors to measure the correlation between the single modality features and the fusion representation. The single modality-fusion representation in the same fusion relationship is regarded as a positive sample pair, and the other single modalities and fusion representations in the same batch are regarded as negative sample pairs. Finally, the contrast loss of the three modalities and the fusion representation is added to form the final contrast loss result, which is used as the emotion recognition prediction result.

[0059] like Figure 4 As shown, the prediction results of the model are shown in the index result Acc-2. It can be seen that as the missing rate increases, the index results of each model decrease. However, the system of the present invention has good robustness. Even in the case of a very high data missing rate, the index results of the model are still at the forefront.

[0060] Embodiment 2: In the second aspect, as Figure 3 As shown, in order to achieve the above-mentioned purpose, the present invention discloses a multimodal emotion recognition system, comprising:

[0061] The data processing module 11 is used to obtain multimodal emotion data, annotate it, and randomly mask the annotated multimodal emotion data to obtain missing emotion data, wherein the multimodal emotion data includes image, voice and text features;

[0062] A feature extraction module 12 is used to input the annotated multimodal sentiment data and missing sentiment data into a pre-established trainable deep neural network model based on a distillation mechanism and contrastive learning, and output fused features and updated unimodal features;

[0063] The recognition prediction module 13 is used to generate positive and negative samples based on the fused features and the updated unimodal features, and input the fused features into the fully connected layer based on the positive and negative samples to obtain the emotion recognition prediction results.

[0064] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0065] It needs to be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0066] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0067] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A multimodal emotion recognition method, characterized in that: The method comprises the following steps: Acquire multimodal emotion data, annotate them, and randomly mask the annotated multimodal emotion data to obtain missing emotion data, wherein the multimodal emotion data includes image, voice, and text features; The annotated multimodal sentiment data and missing sentiment data are input into a pre-established trainable deep neural network model based on distillation mechanism and contrastive learning, and the fused features and updated unimodal features are output; Based on the fused features and the updated unimodal features, positive and negative samples are generated. Based on the positive and negative samples, the fused features are input into the fully connected layer to obtain the emotion recognition prediction results.

2. A multimodal emotion recognition method according to claim 1, characterized in that: The pre-established trainable deep neural network model based on distillation mechanism and contrastive learning includes a teacher network, a student network, a distillation algorithm module and a contrastive prediction coding module.

3. A multimodal emotion recognition method according to claim 2, characterized in that: The construction process of the pre-established trainable deep neural network model based on distillation mechanism and contrastive learning includes the following steps: The complete modality data is used as the input of the teacher network model for training, and the teacher fusion features containing comprehensive multimodal information are obtained, and the pre-training weights are saved; The missing features are used as the input of the student model for training to obtain the student fusion features; Input the teacher fusion features and the student fusion features into the distillation algorithm together; The missing data is then input into the contrastive prediction coding module through the unimodal features of the encoder of the student model and the student fusion features, retaining valuable key task information, and finally generating a trainable deep neural network model based on distillation mechanism and contrastive learning.

4. A multimodal emotion recognition method according to claim 3, characterized in that: The teacher network model includes a feature extraction module, a feature reconstruction module and a late fusion module; The feature extraction module is a feature extraction network based on mutual attention and self-attention, which is stacked by 3 one-dimensional convolutional layers, 3 position embedding layers, 6 cross-modal attention encoders, 6 self-attention encoders and 3 splicing operations. The feature reconstruction module is used to reconstruct the original complete sequence from the sequence generated by the feature extraction module. The late fusion module obtains the teacher fusion result, including 3 bidirectional gated recurrent units, 3 activation functions, 3 one-dimensional convolutional networks, 3 maximum pooling and 1 splicing operation.

5. A multimodal emotion recognition method according to claim 4, characterized in that: The student model consists of a unimodal encoder and a global-local cross-modal fusion module, wherein the unimodal encoder extracts modal discourse representation and time step features, and inputs the features into a linear layer to map them into the same dimension and concatenates the discourse-level representations of the three modalities to obtain a global multimodal representation; the global-local cross-modal fusion module gradually improves the information interaction between each layer in a multi-layer stacking manner, and the time step features from each modality and the global modal representation are subjected to cross-modal attention, self-attention operations and feedforward neural networks; after the multi-layer cross-modal attention modules are stacked, the time dimension of the final global multimodal context shared representation is flattened, and then concatenated with the extracted unimodal representation, mapped to a low-dimensional space through a linear layer to obtain a student fusion result, and finally the fused representation is input into a multi-layer perceptron to output a prediction result.

6. A multimodal emotion recognition method according to claim 5, characterized in that: The distillation algorithm embeds comprehensive multimodal knowledge into the student model to utilize complete modality information to assist learning with missing data.

7. A multimodal emotion recognition method according to claim 6, characterized in that: The contrast prediction coding module uses a scoring function acting on two normalized vectors to measure the correlation between the unimodal features and the fusion representation. The unimodal-fusion representations in the same fusion relationship are regarded as positive sample pairs, and other unimodal and fusion representations in the same batch are regarded as negative sample pairs. Finally, the contrast losses of the three modalities and the fusion representation are added together to form the final contrast loss result.

8. A multimodal emotion recognition method according to claim 7, characterized in that: The modal data with loss in proportion is input into the model, and the output is the probability value of the emotion category. If the emotion polarity result corresponding to the maximum probability is the same as the true emotion category, it means that the prediction is correct, and the classification accuracy of the multimodal emotion recognition is obtained, that is, the proportion of samples with correct prediction to the total samples.

9. A multimodal emotion recognition method according to claim 3, characterized in that: The pre-established trainable deep neural network model based on distillation mechanism and contrastive learning is expressed as follows: KC4RM=T-Net+S-Net+KDM+CLM KC4RM is the general name of the model, T-Net is the teacher network, S-Net is the student network, KDM is the knowledge distillation algorithm module, and CLM is comparative predictive coding.

10. A multimodal emotion recognition system, characterized in that: include: A data processing module, used to obtain multimodal emotion data, annotate it, and randomly mask the annotated multimodal emotion data to obtain missing emotion data, wherein the multimodal emotion data includes image, voice and text features; The feature extraction module is used to input the annotated multimodal sentiment data and missing sentiment data into a pre-established trainable deep neural network model based on distillation mechanism and contrastive learning, and output fused features and updated unimodal features; The recognition prediction module is used to generate positive and negative samples based on the fused features and the updated unimodal features, and input the fused features into the fully connected layer based on the positive and negative samples to obtain the emotion recognition prediction results.

Images