A multi-modal emotion recognition method and system

Abstract

This invention discloses a multimodal emotion recognition method and system, relating to the field of image processing technology, comprising the following steps: acquiring multimodal emotion data and missing emotion data, and labeling them to obtain labeled multimodal emotion data and missing emotion data, wherein the multimodal emotion data includes image, speech, and text features; inputting the labeled multimodal emotion data and missing emotion data into a pre-established trainable deep neural network model based on distillation mechanism and contrastive learning, and outputting fused features and updated single-modal features; generating positive and negative samples based on the fused features and updated single-modal features, and inputting the fused features into a fully connected layer based on the positive and negative samples to obtain the emotion recognition prediction result.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically a multimodal emotion recognition method and system. Background Technology

[0002] Multimodal sentiment analysis has always been a hot research topic in affective computing. People expressing their emotions and opinions through videos on social media has become a trend. Researchers are utilizing the massive amounts of video data generated daily on social media to conduct extensive research on applications such as intelligent human-computer interaction, driver fatigue detection, and family care for the elderly. Multimodal sentiment analysis extracts rich emotional information from visual, speech, and text data, making it more effective than single-modal systems in determining the emotional state of an individual. This is because the data from different modalities complement each other, effectively eliminating the ambiguity inherent in emotions expressed by single modalities. However, in practical applications, modal data is often incomplete. For example, ambient noise can interfere with the speaker's voice, and sensor problems can result in blurry images captured by cameras. These unavoidable factors significantly impact the accuracy of multimodal sentiment recognition.

[0003] Most existing studies assume the absence of only one modality, ignoring the more common scenario of multiple modalities being missing and uncertain. These methods typically employ a generative approach, requiring the training of a new model each time to adapt to each missing modality, which is clearly inflexible. While some studies have used joint representations, addressing the issue of multiple missing modalities, most of these studies fix the missing samples during training, thus limiting their ability to handle more complex real-world scenarios. Summary of the Invention

[0004] To address the shortcomings mentioned in the background section, the present invention aims to provide a multimodal emotion recognition method and system that can resolve the issue of missing random features and improve the predictive performance of network models.

[0005] Firstly, the objective of this invention can be achieved through the following technical solution: a multimodal emotion recognition method, the method comprising the following steps:

[0006] Multimodal sentiment data is acquired and labeled. The labeled multimodal sentiment data is then randomly masked to obtain missing sentiment data. The multimodal sentiment data includes image, speech, and text features.

[0007] The labeled multimodal sentiment data and missing sentiment data are input into a pre-built trainable deep neural network model based on distillation mechanism and contrastive learning, and the output is fused features and updated single-modal features.

[0008] Based on the fused features and the updated single-modal features, positive and negative samples are generated. The fused features are then input into the fully connected layer based on the positive and negative samples to obtain the emotion recognition prediction results.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the pre-established trainable deep neural network model based on distillation mechanism and contrastive learning includes a teacher network, a student network, a distillation algorithm module, and a contrastive prediction encoding module.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the construction process of the pre-established trainable deep neural network model based on distillation mechanism and contrastive learning includes the following steps:

[0011] The complete modal data is used as input to train the teacher network model to obtain teacher fusion features containing comprehensive multimodal information, and the pre-trained weights are saved.

[0012] The missing features are used as input to train the student model to obtain the student fused features;

[0013] The teacher fusion features and student fusion features are both input into the distillation algorithm;

[0014] The missing data is then fed into the contrastive prediction coding module by the single-modal features of the encoder of the student model and the fused features of the student model, retaining valuable key task information, and finally generating a trainable deep neural network model based on distillation mechanism and contrastive learning.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the teacher network model includes a feature extraction module, a feature reconstruction module, and a post-fusion module; wherein the feature extraction module is a feature extraction network based on mutual attention and self-attention, consisting of 3 one-dimensional convolutional layers, 3 position embedding layers, 6 cross-modal attention encoders, 6 self-attention encoders, and 3 concatenation operations stacked together; the feature reconstruction module is used to reconstruct the original complete sequence from the sequence generated by the feature extraction module; and the post-fusion module obtains the teacher fusion result, including 3 bidirectional gated recurrent units, 3 activation functions, 3 one-dimensional convolutional networks, 3 max pooling operations, and 1 concatenation operation.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the student model consists of a unimodal encoder and a global-local cross-modal fusion module, wherein the unimodal encoder extracts modal discourse representations and time step features, and inputs the features into a linear layer to map them to the same dimension, and concatenates the discourse-level representations of the three modalities to obtain a global multimodal representation; the global-local cross-modal fusion module adopts a multi-layer stacking approach to gradually improve the information interaction between each layer, and the time step features from each modality and the global modal representation undergo cross-modal attention, self-attention operations, and a feedforward neural network; after the multi-layer cross-modal attention modules are stacked, the time dimension of the final global multimodal context-shared representation is flattened, and then concatenated with the extracted unimodal representation, and mapped to a low-dimensional space through a linear layer to obtain the student fusion result, and finally the fused representation is input into a multilayer perceptron to output the prediction result.

[0017] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the distillation algorithm uses comprehensive multimodal knowledge to assist in learning with missing data by embedding it into the student model, thereby utilizing complete modal information.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the contrastive prediction encoding module uses a scoring function acting on two normalized vectors to measure the correlation between single-modal features and fused representations, single-modal-fused representations in the same fusion relationship are regarded as positive sample pairs, other single-modal features and fused representations in the same batch are regarded as negative sample pairs, and the contrastive losses of the last three modalities and fused representations are added together to form the final contrastive loss result.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: inputting the modal data that is lost proportionally into the model, and outputting the probability value of the emotion category. If the emotion polarity result corresponding to the highest probability is the same as the true emotion category, it indicates that the prediction is correct, and the classification accuracy of the multimodal emotion recognition is obtained, that is, the proportion of correctly predicted samples to the total samples.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the pre-established trainable deep neural network model based on distillation mechanism and contrastive learning is expressed as follows:

[0021] KC4RM = T-Net + S-Net + KDM + CLM

[0022] KC4RM is the general name for this model, T-Net is the teacher network, S-Net is the student network, KDM is the knowledge distillation algorithm module, and CLM is the contrastive predictive coding.

[0023] Secondly, in order to achieve the above objectives, the present invention discloses a multimodal emotion recognition system, comprising:

[0024] The data processing module is used to acquire multimodal sentiment data, label it, and randomly mask the labeled multimodal sentiment data to obtain missing sentiment data. The multimodal sentiment data includes image, speech, and text features.

[0025] The feature extraction module is used to input the labeled multimodal sentiment data and missing sentiment data into a pre-built trainable deep neural network model based on distillation mechanism and contrastive learning, and output fused features and updated single-modal features.

[0026] The recognition and prediction module generates positive and negative samples based on the fused features and the updated single-modal features. Based on the positive and negative samples, the fused features are input into the fully connected layer to obtain the emotion recognition and prediction results.

[0027] The beneficial effects of this invention are:

[0028] This invention effectively utilizes the characteristics of the knowledge distillation transfer mechanism to assist the training of the student network by transferring comprehensive modal information from the teacher network to address learning with missing data. Positive and negative sample pairs are constructed between the fused features and single-modal features of the student network, retaining valuable task-relevant information to better enhance the correlation between fused and single-modal features. In summary, the distillation mechanism and contrastive learning network proposed in this invention improve the accuracy of sentiment recognition in real-world scenarios with missing data, which is beneficial to the future development of the field of sentiment computing. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0031] Figure 2 This is a schematic diagram of the workflow of the present invention;

[0032] Figure 3 This is a schematic diagram of the system structure of the present invention;

[0033] Figure 4 This is an experimental result diagram of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Example 1:

[0036] The following is a description of the relevant terms used in the embodiments of this application:

[0037] Multimodal features: Multimodal features refer to features extracted from multiple heterogeneous modalities of data. These features can come from different data sources, such as images, text, and speech. The importance of multimodal features lies in their ability to provide more comprehensive and richer information, helping to improve the accuracy of recognition and classification. For example, in biometrics, multimodal technology can improve the accuracy and security of authentication by fusing multiple biometric features such as fingerprints, faces, and irises.

[0038] like Figure 1 As shown, a multimodal emotion recognition method includes the following steps:

[0039] S101: Acquire multimodal sentiment data and label it. Randomly mask the labeled multimodal sentiment data to obtain missing sentiment data. The multimodal sentiment data includes image, speech and text features.

[0040] The raw modal data requires preprocessing. For text modalities, a pre-trained BERT model extracts features with a dimension of 768. For international datasets, COVAREP is used to extract various low-level audio features, and Facet is used to extract facial expression features. For Chinese datasets, Librosa is used to extract audio features, and OpenFace is used to extract facial units. To capture real-world scenarios with random feature loss, a masking function is used to randomly mask the complete sequence, setting the masked features to 0.

[0041] S102: Input the labeled multimodal sentiment data and missing sentiment data into a pre-established trainable deep neural network model based on distillation mechanism and contrastive learning, and output the fused features and updated single-modal features.

[0042] The pre-built trainable deep neural network model based on distillation mechanism and contrastive learning includes a teacher network, a student network, a distillation algorithm module, and a contrastive prediction encoding module.

[0043] The process of constructing a pre-built trainable deep neural network model based on distillation mechanism and contrastive learning includes the following steps:

[0044] The complete modal data is used as input to train the teacher network model to obtain teacher fusion features containing comprehensive multimodal information, and the pre-trained weights are saved.

[0045] The missing features are used as input to train the student model to obtain the student fused features;

[0046] The teacher fusion features and student fusion features are both input into the distillation algorithm;

[0047] The missing data is then fed into the contrastive prediction coding module by the single-modal features of the encoder of the student model and the fused features of the student model, retaining valuable key task information, and finally generating a trainable deep neural network model based on distillation mechanism and contrastive learning.

[0048] The teacher network model includes a feature extraction module, a feature reconstruction module, and a post-fusion module. The feature extraction module is a feature extraction network based on mutual attention and self-attention, consisting of three one-dimensional convolutional layers, three position embedding layers, six cross-modal attention encoders, six self-attention encoders, and three concatenation operations stacked together. The feature reconstruction module is used to reconstruct the original complete sequence from the sequence generated by the feature extraction module. The post-fusion module obtains the teacher fusion result, which includes three bidirectional gated recurrent units, three activation functions, three one-dimensional convolutional networks, three max pooling operations, and one concatenation operation.

[0049] The student model consists of a unimodal encoder and a global-local cross-modal fusion module. The unimodal encoder extracts modal discourse representations and temporal step features, and inputs these features into a linear layer to map them to the same dimension. The discourse-level representations of the three modalities are then concatenated to obtain a global multimodal representation. The global-local cross-modal fusion module uses a multi-layer stacking approach to progressively improve the information interaction between each layer. The temporal step features from each modality and the global modal representation undergo cross-modal attention, self-attention operations, and a feedforward neural network. After stacking multiple cross-modal attention modules, the temporal dimension of the final global multimodal context-shared representation is flattened and then concatenated with the extracted unimodal representation. This is then mapped to a low-dimensional space through a linear layer to obtain the student fusion result. Finally, the fused representation is input into a multilayer perceptron to output the prediction result.

[0050] The distillation algorithm utilizes complete modal information to assist in learning when data is missing by embedding comprehensive multimodal knowledge into the student model.

[0051] The contrastive prediction encoding module uses a scoring function that operates on two normalized vectors to measure the correlation between single-modal features and fused representations. Single-modal features and fused representations with the same fusion relationship are considered positive sample pairs, while other single-modal features and fused representations in the same batch are considered negative sample pairs. Finally, the contrastive losses of the three modal features and fused representations are added together to form the final contrastive loss result.

[0052] The modal data that is lost proportionally is input into the model, and the output is the probability value of the sentiment category. If the sentiment polarity result corresponding to the highest probability is the same as the actual sentiment category, it means that the prediction is correct, and the classification accuracy of this multimodal sentiment recognition is obtained, that is, the proportion of correctly predicted samples to the total samples.

[0053] The pre-established trainable deep neural network model based on distillation mechanism and contrastive learning is expressed as follows:

[0054] KC4RM = T-Net + S-Net + KDM + CLM

[0055] In this model, KC4RM is the general term, T-Net is the teacher network, S-Net is the student network, KDM is the knowledge distillation algorithm module, and CLM is contrastive prediction coding. Specifically, the following examples further illustrate the solution of this invention: Figure 2 As shown, this deep neural network model consists of a teacher network, a student network, knowledge distillation, and a contrastive prediction encoding module. The teacher model is pre-programmed with complete modality data. Training, where T m It is the sequence length, d m This refers to the feature dimension of modality m∈{t,v,a}. The entire model in the second stage uses missing modalities as input. The teacher network encodes modalities of different sequence lengths into the same dimension, obtaining updated single-modal representations, which are then reconstructed and fused with the original features to obtain the final result. The distillation algorithm module is based on relational consistency to transfer teacher knowledge; its input is the teacher fusion F. t -Student's fusion layer representation F s The student network focuses on deep interactions between missing modalities, utilizing the discourse-level context of each modality as the global multimodal context, and performing cross-modal attention with local unimodal features. Simultaneously, a contrastive learning strategy is introduced into the student network to maximize mutual information between the fused result and the input unimodality, i.e., minimizing the contrastive learning loss function.

[0056] The first training stage uses complete modality features as input to the teacher network, and the teacher's fusion result is obtained through a post-fusion process. The second training stage uses missing features as input to the student network. During training, a distillation loss function is used to help the missing modalities learn the semantics of the complete modality. In addition, a contrastive learning loss function is added to address the problem of information redundancy from the input to the fusion layer.

[0057] S103: Based on the fused features and the updated single-modal features, positive and negative samples are generated. Based on the positive and negative samples, the fused features are input into the fully connected layer to obtain the emotion recognition prediction result.

[0058] During student model training, task-related information is injected into the fusion result. The goal is to ensure that the fused representation retains as much of the most important parts of each unimodality as possible. A scoring function acting on two normalized vectors is used to measure the correlation between unimodal features and the fused representation. Unimodal-fused representations with the same fusion relationship are considered positive sample pairs, while other unimodal-fused representations in the same batch are considered negative sample pairs. Finally, the contrastive losses of the three modalities and the fused representation are summed to form the final contrastive loss result, which serves as the sentiment recognition prediction result.

[0059] like Figure 4 As shown, the prediction results of the model are represented by the index results Acc-2. It can be seen that as the missing data rate increases, the index results of all models decrease. However, the system of this invention has good robustness; even with a very high data missing rate, the model's index results still rank among the top.

[0060] Example 2: Second aspect, such as Figure 3 As shown, in order to achieve the above objectives, this invention discloses a multimodal emotion recognition system, comprising:

[0061] The data processing module 11 is used to acquire multimodal sentiment data and label it, and randomly mask the labeled multimodal sentiment data to obtain missing sentiment data. The multimodal sentiment data includes image, speech and text features.

[0062] The feature extraction module 12 is used to input the labeled multimodal sentiment data and missing sentiment data into a pre-established trainable deep neural network model based on distillation mechanism and contrastive learning, and output fused features and updated single-modal features.

[0063] The recognition and prediction module 13 is used to generate positive and negative samples based on the fused features and the updated single-modal features, and input the fused features into the fully connected layer based on the positive and negative samples to obtain the emotion recognition and prediction results.

[0064] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0065] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0066] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0067] The foregoing has shown and described the basic principles, main features, and advantages of this disclosure. Those skilled in the art should understand that this disclosure is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this disclosure. Various changes and modifications can be made to this disclosure without departing from its spirit and scope, and all such changes and modifications fall within the scope of this disclosure as claimed.

Claims

1. A multimodal emotion recognition method, characterized in that, The method includes the following steps: Multimodal sentiment data is acquired and labeled. The labeled multimodal sentiment data is then randomly masked to obtain missing sentiment data. The multimodal sentiment data includes image, speech, and text features. The labeled multimodal sentiment data and missing sentiment data are input into a pre-built trainable deep neural network model based on distillation mechanism and contrastive learning, and the output is fused features and updated single-modal features. The pre-established trainable deep neural network model based on distillation mechanism and contrastive learning includes a teacher network, a student network, a distillation algorithm module, and a contrastive prediction coding module. The process of constructing the pre-established trainable deep neural network model based on distillation mechanism and contrastive learning includes the following steps: The complete modal data is used as input to train the teacher network model to obtain teacher fusion features containing comprehensive multimodal information, and the pre-trained weights are saved. The missing features are used as input to train the student model to obtain the student fused features; The teacher fusion features and student fusion features are both input into the distillation algorithm; The missing data is then fed into the contrastive prediction coding module by the single-modal features of the encoder of the student model and the fused features of the student, retaining valuable key task information, and finally generating a trainable deep neural network model based on distillation mechanism and contrastive learning. The distillation algorithm embeds comprehensive multimodal knowledge into the student model to assist in learning with missing data by utilizing complete modal information. The distillation algorithm is based on relational consistency to transfer teacher knowledge, and its input is the teacher's fusion... -Student's fusion layer representation ; The contrastive prediction encoding module uses a scoring function that operates on two normalized vectors to measure the correlation between single-modal features and fused representations. Single-modal-fused representations with the same fusion relationship are regarded as positive sample pairs, and other single-modal features and fused representations in the same batch are regarded as negative sample pairs. Finally, the contrastive losses of the three modalities and fused representations are added together to form the final contrastive loss result. Based on the fused features and the updated single-modal features, positive and negative samples are generated. The fused features are then input into the fully connected layer based on the positive and negative samples to obtain the emotion recognition prediction results.

2. The multimodal emotion recognition method according to claim 1, characterized in that, The teacher network model includes a feature extraction module, a feature reconstruction module, and a post-fusion module; The feature extraction module is based on a mutual attention and self-attention feature extraction network, consisting of three one-dimensional convolutional layers, three position embedding layers, six cross-modal attention encoders, six self-attention encoders, and three concatenation operations stacked together. The feature reconstruction module is used to reconstruct the original complete sequence from the sequence generated by the feature extraction module. The post-fusion module obtains the teacher fusion result, including three bidirectional gated recurrent units, three activation functions, three one-dimensional convolutional networks, three max pooling operations, and one concatenation operation.

3. The multimodal emotion recognition method according to claim 2, characterized in that, The student model consists of a unimodal encoder and a global-local cross-modal fusion module. The unimodal encoder extracts modal discourse representations and temporal step features, and maps these features to the same dimension through a linear layer. The discourse-level representations of the three modalities are then concatenated to obtain a global multimodal representation. The global-local cross-modal fusion module uses a multi-layer stacking approach to progressively improve the information interaction between each layer. The temporal step features from each modality and the global modal representation undergo cross-modal attention, self-attention operations, and a feedforward neural network. After stacking multiple cross-modal attention modules, the temporal dimension of the final global multimodal context-shared representation is flattened and then concatenated with the extracted unimodal representation. This is then mapped to a low-dimensional space through a linear layer to obtain the student fusion result. Finally, the fused representation is input into a multilayer perceptron to output the prediction result.

4. The multimodal emotion recognition method according to claim 3, characterized in that, The modal data that is lost proportionally is input into the model, and the output is the probability value of the sentiment category. If the sentiment polarity result corresponding to the highest probability is the same as the actual sentiment category, it means that the prediction is correct, and the classification accuracy of this multimodal sentiment recognition is obtained, that is, the proportion of correctly predicted samples to the total samples.

5. The multimodal emotion recognition method according to claim 1, characterized in that, The pre-established trainable deep neural network model based on distillation mechanism and contrastive learning is expressed as follows: KC4RM = T-Net + S-Net + KDM + CLM In this model, KC4RM is the general term for the model, T-Net is the teacher network, S-Net is the student network, KDM is the knowledge distillation algorithm module, and CLM is the contrastive predictive coding.

6. A multimodal emotion recognition system, employing the multimodal emotion recognition method according to any one of claims 1 to 5, characterized in that, include: The data processing module is used to acquire multimodal sentiment data, label it, and randomly mask the labeled multimodal sentiment data to obtain missing sentiment data. The multimodal sentiment data includes image, speech, and text features. The feature extraction module is used to input the labeled multimodal sentiment data and missing sentiment data into a pre-built trainable deep neural network model based on distillation mechanism and contrastive learning, and output fused features and updated single-modal features. The recognition and prediction module generates positive and negative samples based on the fused features and the updated single-modal features. Based on the positive and negative samples, the fused features are input into the fully connected layer to obtain the emotion recognition and prediction results.

Images