Multimodal emotion recognition method, device, equipment, computer storage medium

Abstract

The application discloses a multimodal emotion recognition method, device and equipment and a computer storage medium, and is applied to the technical field of intelligent recognition, and specifically comprises the following steps: extracting labeled information of different modes from labeled video data; training the feature fusion model to obtain a trained feature fusion model; extracting unlabeled information of different modes from unlabeled video data; processing the unlabeled features through the trained feature fusion model; processing the labeled data and selected pseudo-labeled data through the trained feature fusion model to obtain a final recognition result and outputting the final recognition result. The above method fully utilizes information of labeled data and unlabeled data, extracts multimodalities through a pre-training model, and effectively combines the features through a feature fusion model, so that the accuracy of emotion classification can be improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent recognition technology, and in particular to multimodal emotion recognition methods, devices, equipment, and computer storage media. Background Technology

[0002] Currently, emotion recognition technology has been widely researched and applied, mainly including text sentiment analysis, speech emotion recognition, and image sentiment analysis.

[0003] Text sentiment analysis refers to the use of computer technology to analyze the emotional information contained in text, thereby identifying the emotional state of the text's author. Currently, the main techniques for text sentiment analysis include sentiment lexicon methods, machine learning, and deep learning. Deep learning can automatically extract features from text and classify it. Because deep learning can consider context, syntax, and semantics, it performs better in sentiment classification. However, deep learning models require significant computational resources and datasets, the training process is complex, and the quality and coverage of the dataset can affect the accuracy of sentiment analysis. For texts with unclear or difficult-to-identify contexts, the accuracy of sentiment analysis will decrease.

[0004] Speech emotion recognition refers to identifying a speaker's emotional state by analyzing the vocal features of a speech signal. This involves extracting features such as tone, pitch, speech rate, and volume from the speech signal to determine the speaker's emotional state. However, these features are influenced by individual speaker differences and environmental noise. Researchers have utilized deep neural networks to model and classify speech signals, automatically extracting features and training models to identify different emotional states. Deep learning, by considering multiple aspects of speech signal information, performs better in emotion recognition. However, limitations in dataset quality can affect the performance and generalization ability of this method.

[0005] Image sentiment analysis refers to identifying the emotional state expressed by an image by analyzing its visual features. This involves extracting features such as color, texture, shape, and spatial structure to determine the emotional state. Traditional methods have low accuracy because visual features are easily affected by factors such as lighting, angle, and resolution. Deep neural networks, however, can automatically extract features and train models to identify different emotional states. Since deep learning considers multiple aspects of image information, it performs better in sentiment analysis. Furthermore, deep learning-based methods can incorporate transfer learning and data augmentation techniques to improve model generalization and data efficiency. However, this approach has limitations. First, emotional information is often implicit, so directly inferring emotions from images can be influenced by subjective factors and uncertainty. Second, emotional expression is often affected by context and individual differences, increasing the complexity of sentiment analysis. Finally, deep learning models require a large amount of labeled sentiment data during training, but obtaining accurate sentiment labels is a difficult and time-consuming task.

[0006] In related academic literature and research, several papers have explored the challenges and limitations of image sentiment analysis. For example, a study titled "Deep Visual-Semantic Alignments for Generating Image Descriptions" (paper: Karpathy et al., 2015) points out that accurately extracting sentiment information from images requires in-depth analysis of the image's semantics and context. The authors note that sentiment information is often interrelated with the semantic and visual features of an image; therefore, both local and global semantic information need to be considered in sentiment analysis. For instance, a person's facial expression may provide clues to their emotions, but the context in which that person is present (e.g., are they attending a party or a funeral) might alter the interpretation of that expression. This necessitates that our models understand the semantics and context of an image, which is a very challenging task.

[0007] Current emotion recognition technologies suffer from several limitations. For example, in speech emotion recognition, the characteristics of speech signals are easily affected by individual speaker differences and environmental noise, impacting accuracy. In text emotion analysis, models cannot account for the influence of context and syntactic structure, resulting in low accuracy. In image emotion analysis, emotional information in images is often implicit, requiring in-depth semantic and contextual analysis. Multimodal data collection and processing of multiple information types, including speech, images, and text, is costly. Consequently, the quality and scale of current multimodal emotion recognition datasets are limited, failing to meet the needs of large-scale applications. Furthermore, compatibility issues exist between different modalities, preventing direct integration of models and algorithms, thus restricting the application scope of multimodal emotion recognition. Summary of the Invention

[0008] Therefore, it is necessary to provide a multimodal emotion recognition method, device, computer equipment, and storage medium to address the aforementioned technical problems, in order to solve the problems of high cost, limited data quality and scale of existing multimodal emotion recognition technologies, inability to meet the needs of large-scale application scenarios, and incompatibility between different modal emotion recognition technologies.

[0009] A multimodal emotion recognition method includes:

[0010] Information on different labeled modalities is extracted from labeled video data, and the information on the different labeled modalities is processed by a corresponding pre-trained model to extract labeled features.

[0011] Using the labeled features as input to the feature fusion model, the feature fusion model is trained to obtain a trained feature fusion model.

[0012] Extract information about different unlabeled modalities from unlabeled video data, and process the information about different unlabeled modalities using a corresponding pre-trained model to extract unlabeled features;

[0013] The unlabeled features are processed by the trained feature fusion model to generate pseudo-labels. Features with higher confidence are selected from the pseudo-labels and added to the training set along with the features with higher confidence and the corresponding pseudo-labels.

[0014] The labeled data and selected pseudo-label data are processed by a pre-trained feature fusion model to obtain the final recognition result, which is then output.

[0015] A multimodal emotion recognition device, comprising:

[0016] The labeled feature extraction unit is used to extract information on different labeled modalities from labeled video data, and to process the information on the different labeled modalities through a corresponding pre-trained model to extract labeled features.

[0017] The model training unit is used to train the feature fusion model by taking the labeled features as input to obtain the trained feature fusion model.

[0018] The unlabeled feature extraction unit is used to extract information on different unlabeled modalities from unlabeled video data, and to process the information on the different unlabeled modalities through a corresponding pre-trained model to extract unlabeled features.

[0019] An unlabeled feature selection unit is used to process the unlabeled features using a pre-trained feature fusion model to generate pseudo-labels, select features with higher confidence from the pseudo-labels, and add the features with higher confidence and the corresponding pseudo-labels to the training set; and

[0020] The fusion recognition unit is used to process the labeled data and the selected pseudo-label data through a pre-trained feature fusion model to obtain the final recognition result and output it.

[0021] A computer device includes a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the above-described multimodal emotion recognition method when executing the computer-readable instructions.

[0022] One or more readable storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the multimodal emotion recognition method described above.

[0023] The aforementioned multimodal emotion recognition methods, devices, computer equipment, and storage media fully utilize information from both labeled and unlabeled data. By extracting multimodal features through a pre-trained model and effectively combining these features through a feature fusion model, the accuracy of emotion classification can be improved. At the same time, by expanding the training set with pseudo-labels, the robustness of the model is further enhanced. Furthermore, through cross-modal interaction and feature dimensionality reduction, the accuracy and robustness of feature representation are improved. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating a multimodal emotion recognition method according to an embodiment of the present invention;

[0026] Figure 2 This is a flowchart illustrating a multimodal emotion recognition method according to an embodiment of the present invention;

[0027] Figure 3 This is a flowchart illustrating a multimodal emotion recognition method according to an embodiment of the present invention;

[0028] Figure 4 This is a flowchart illustrating a multimodal emotion recognition method according to an embodiment of the present invention;

[0029] Figure 5 This is a schematic diagram of the overall process of the multimodal emotion recognition method in Chinese according to an embodiment of the present invention;

[0030] Figure 6 This is a schematic diagram of the overall process of the multimodal emotion recognition method in one embodiment of the present invention.

[0031] Figure 7 This is a flowchart illustrating a multimodal emotion recognition method according to an embodiment of the present invention;

[0032] Figure 8 This is a schematic diagram of the overall process of fusing labeled data and selected pseudo-label data in a multimodal emotion recognition method according to an embodiment of the present invention.

[0033] Figure 9 This is a schematic diagram illustrating the overall process of fusing labeled data and selected pseudo-labeled data in a multimodal emotion recognition method according to an embodiment of the present invention.

[0034] Figure 10 This is a schematic diagram of a multimodal emotion recognition device according to an embodiment of the present invention;

[0035] Figure 11 This is a schematic diagram of a computer device according to an embodiment of the present invention;

[0036] Figure 12 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] In one embodiment, such as Figure 1 As shown, a multimodal emotion recognition method is provided, including the following steps:

[0039] In step S110, information on different labeled modalities is extracted from the labeled video data, and the information on the different labeled modalities is processed by the corresponding pre-trained model to extract labeled features.

[0040] In the embodiments of this application, the information in different modalities includes video frames, text, and audio, wherein the video frame can be a frame image, the text can be dialogue, and the audio can be speech.

[0041] In practical applications, the amount of labeled video data can be small. Information on different modalities can be extracted from this labeled video data and input into the corresponding pre-trained models to extract labeled features.

[0042] In the embodiments of this application, the corresponding pre-trained models provide different pre-trained models for information of different modalities. For example, for audio, the HuBERT model is used as the pre-trained model, and for text, the MacBERT model is used as the pre-trained model.

[0043] In step S120, the labeled features are used as input to the feature fusion model to train the feature fusion model and obtain a trained feature fusion model.

[0044] In the embodiments of this application, the feature fusion model aims to effectively combine features from different modalities, thereby improving the accuracy of emotion classification.

[0045] In step S130, information on different unlabeled modalities is extracted from the unlabeled video data, and the information on the different unlabeled modalities is processed by the corresponding pre-trained model to extract unlabeled features.

[0046] In practical applications, labeled video data is often limited, while unlabeled video data is often abundant, serving as the target data source for multimodal emotion recognition methods. Its advantage lies in fully utilizing the information from both the limited amount of labeled data and the large amount of unlabeled data, further enhancing the model's robustness.

[0047] In step S140, the unlabeled features are processed by the trained feature fusion model to generate pseudo-labels. Features with higher confidence are selected from the pseudo-labels, and the features with higher confidence and the corresponding pseudo-labels are added to the training set.

[0048] In step S150, the labeled data and the selected pseudo-label data are processed by the trained feature fusion model to obtain the final recognition result and output it.

[0049] The aforementioned multimodal emotion recognition methods, devices, computer equipment, and storage media fully utilize information from both labeled and unlabeled data. By extracting multimodal data through a pre-trained model and effectively combining these features through a feature fusion model, the accuracy of emotion classification can be improved. Furthermore, by expanding the training set using pseudo-labels, the robustness of the model is further enhanced.

[0050] In one embodiment, such as Figure 2 As shown, a multimodal emotion recognition method is provided. This method extracts different frames from a video as image frames, combining static and dynamic information about a person's emotions. To this end, an innovative expMAE (Expression MAE) is introduced. Based on the idea of ​​a mask autoencoder, it introduces two branches during the recognition process to process the static and dynamic information of the video respectively. Specifically, the steps for extracting labeled information of different modalities from labeled video data, or the steps for extracting unlabeled information of different modalities from unlabeled video data, include the following steps:

[0051] In step S111, a frame of the video is randomly selected as the input image for the current training round, and a random masking operation and restoration operation are performed on the single image to capture the static features of facial expressions in each frame.

[0052] In step S112, multiple video frames are used as input simultaneously, and random masking and restoration operations are performed to extract dynamic features of facial expressions in the video.

[0053] In this embodiment, to improve the efficiency of masking autoencoding for multiple video frames, in the step of simultaneously using multiple video frames as input and performing random masking and restoration operations to extract dynamic features of facial expressions in the video, the random masking operation specifically involves maintaining consistent mask regions across different frames, i.e., employing a pipeline masking strategy. Through this method, we can effectively extract dynamic features of facial expressions in the video while obtaining more useful and complementary visual modal features.

[0054] The multimodal emotion recognition method provided in this application uses the expMAE model, which is designed to process both static and dynamic information from videos simultaneously. By combining static and dynamic features, the model can acquire more comprehensive and richer visual modal features. This comprehensive feature extraction method enables our model to more accurately capture changes and expressions of human emotions, further improving the performance and robustness of emotion recognition.

[0055] In one embodiment, such as Figure 3 As shown, a multimodal emotion recognition method is provided, wherein the information of different modalities includes audio. The steps of extracting labeled information of different modalities from labeled video data and processing the labeled information of different modalities through a corresponding pre-trained model to extract labeled features include:

[0056] In step S113, labeled audio is extracted from the labeled video data and the labeled audio is converted into a labeled Mel spectrogram.

[0057] In step S114, the labeled Mel spectrogram is processed using the HuBERT model to extract labeled audio features.

[0058] The step of extracting unlabeled information of different modalities from unlabeled video data, and processing the information of the different unlabeled modalities through a corresponding pre-trained model to extract unlabeled features, includes:

[0059] In step S115, unlabeled audio is extracted from the unlabeled video data and the unlabeled audio is converted into an unlabeled Mel spectrogram.

[0060] In step S116, the unlabeled Mel spectrogram is processed using the HuBERT model to extract unlabeled audio features.

[0061] In this embodiment, HuBERT (Hybrid Unsupervised and Supervised Representations for Transformers) is an innovative model with prior audio knowledge, specifically designed for audio processing. Its design philosophy combines unsupervised and supervised learning methods to extract high-quality audio representations. The model employs a two-stage training approach: pre-training and fine-tuning. During pre-training, HuBERT utilizes large-scale unlabeled audio data for self-supervised learning. Specifically, the model performs two unsupervised learning tasks: mask prediction and contrastive learning. In the mask prediction task, the model predicts the masked portions of the input audio segment, while the contrastive learning task requires the model to learn good audio representations by comparing pairs of audio segments with similar semantics. This combination of unsupervised learning tasks enables HuBERT to learn rich audio features from large-scale data, fully leveraging prior audio knowledge. Furthermore, HuBERT employs a hybrid embedding strategy, embedding audio features into multiple different representation spaces, thereby providing more comprehensive and diverse audio modal features. This strategy further enhances the HuBERT model's ability to represent audio data and its diversity of representations.

[0062] In one embodiment, such as Figure 4 As shown, a multimodal emotion recognition method is provided, wherein the information of different modalities includes text. The steps of extracting labeled information of different modalities from labeled video data and processing the labeled information of different modalities through a corresponding pre-trained model to extract labeled features include:

[0063] In step S117, tagged text is extracted from the tagged video data.

[0064] In step S118, the labeled text is processed using the MacBERT model to extract labeled text features.

[0065] The step of extracting unlabeled information of different modalities from unlabeled video data, and processing the information of the different unlabeled modalities through a corresponding pre-trained model to extract unlabeled features, includes:

[0066] In step S119, unlabeled text is extracted from the unlabeled video data.

[0067] In step S1110, the labeled text is processed using the MacBERT model to extract labeled text features.

[0068] In this embodiment of the application, the text can be dialogue extracted from a video, which can be used as input to the large text model to extract text modal features.

[0069] In this embodiment, MacBERT (MLM as correction BERT) is an improved version of BERT, introducing a pre-training task for error-correcting masked language models (MLM as correction, Mac) to alleviate the inconsistency between pre-training and downstream tasks. In masked language models (MLM), a [MASK] marker is used for masking, but this marker does not appear in the downstream task. Therefore, MacBERT uses n-gram masking and similar words to replace [MASK]. This word-segmentation masking method combined with synonym replacement effectively solves the inconsistency between upstream and downstream tasks, where there are mask markers in the pre-training stage but no mask in the downstream task. When no similar words are available for replacement, random words are used. This is the main improvement of MacBERT; the other major frameworks of MacBERT are completely consistent with BERT.

[0070] In this embodiment of the application, considering that the dialogue extracted from video clips lacks context and background information, relying solely on the text features extracted from the dialogue cannot effectively perform emotion recognition, the information of different modalities includes text and image frames. The step of extracting labeled information of different modalities from labeled video data includes:

[0071] Cross-features between text and image frames are extracted using the CLIP multimodal model.

[0072] In the embodiments of this application, CLIP (Contrastive Language–Image Pre-training) is a pre-training method or model based on contrastive text-image pairs.

[0073] The multimodal emotion recognition method provided in this application creatively utilizes cross-modal features, extracting cross features through the multimodal model CLIP to compensate for the shortcomings of text features. By combining the features extracted by CLIP with text modal features, this method can obtain more comprehensive and accurate emotion information.

[0074] Figure 5 , Figure 6 The overall flow of the multimodal emotion recognition method provided in this application is illustrated in both Chinese and English diagrams.

[0075] In one embodiment of this application, reference is made to Figure 7The multimodal emotion recognition method provided in this application includes labeled data and selected pseudo-labeled data, which respectively include dynamic features, static features, text features, audio features, and cross-features of text and image frames. The step of processing the labeled data and selected pseudo-labeled data through a trained feature fusion model to obtain the final recognition result includes:

[0076] In step S151, the dynamic and static features are processed by a bilinear pooling model to obtain the fused visual features.

[0077] In the embodiments of this application, the static and dynamic features extracted by ExpressionMAE are used as inputs to the bilinear pooling model, and the dimension of the input feature vectors is 1*1024. First, the two input features are multiplied by a dot product, that is, the corresponding elements are multiplied one by one. This can capture the interaction relationship between the two features. Then, a Sum Pooling operation is performed to add all elements to obtain a scalar value. Finally, the result of Sum Pooling is L2 regularized to output the fused features.

[0078] In step S152, the cross features and text features of the text and image frames are processed by a bilinear pooling model to obtain fused visual text features.

[0079] In step S153, the visual features, visual text features, and audio features are processed by the Attention model to obtain the final fused feature representation.

[0080] In this application, the final fusion is performed at the feature level through an attention mechanism.

[0081] In step S154, the final feature representation is classified into emotions through a linear layer to obtain the final recognition result.

[0082] The multimodal emotion recognition method provided in this application uses a Factorized Bilinear Pooling (FBP) model for feature fusion. The core idea of ​​FBP is to perform bilinear projection and decomposition on the input features, allowing features from different modalities to interact and generating a fused feature representation. FBP can effectively capture the interaction information between different modalities through bilinear projection and decomposition. By multiplying features from different modalities, FBP can capture the nonlinear relationships between modalities, thus better fusing information from different modalities. By decomposing the input features, high-dimensional feature representations can be mapped to a low-dimensional space. This dimensionality reduction reduces feature redundancy and improves feature expressiveness and computational efficiency. Furthermore, FBP retains important cross-modal interaction information during feature fusion. By performing bilinear projection on the input features, correlations between modalities can be captured while filtering out irrelevant information, thereby improving the accuracy and robustness of feature representation. Most importantly, FBP is applicable to a variety of different modalities, including images, text, and audio. It can handle features of different dimensions and types and effectively fuse them together.

[0083] Figure 8 , Figure 9 The overall process of fusing labeled data and selected pseudo-labeled data by the multimodal emotion recognition method provided in this application is illustrated in Chinese and English diagrams, respectively.

[0084] This patent proposes a multimodal emotion recognition method. First, it divides video data into three modalities: visual (image frames), audio (speech), and text (dialogue). This decomposition fully utilizes the various representations within the video data, thus capturing emotional information more comprehensively. Then, multiple pre-trained models with prior knowledge are used to extract features from the different modalities. Finally, the extracted features are fused to obtain the emotion recognition result. The fused features comprehensively utilize information from different modalities, improving the accuracy and robustness of emotion recognition.

[0085] This patent application proposes an innovative multimodal emotion recognition method. It uses a pre-trained expMAE model to extract modal features that simultaneously contain static and dynamic information. It utilizes large pre-trained models with prior knowledge of audio and text to extract diverse audio modal features and text modal features respectively. Furthermore, it innovatively uses the CLIP model to extract cross features, thereby enhancing the connection between different modalities.

[0086] The method in this patent is primarily trained and tested on the MER2023 dataset. Due to the high cost of video annotation, it is difficult to collect a large number of samples with emotion labels. Therefore, the official staff of MER2023 extracted human-centered video clips containing emotional expressions from movies and TV series, dividing them into the MER-SEMI subset, which includes over 70,000 unlabeled data points and over 800 labeled data points. The official video emotion recognition results baseline of the MER2023 dataset are shown in Table 1.

[0087] Table 1 MER2023 baseline

[0088]

[0089] As shown in Table 2, the visual modal extracted by the proposed expMAE method in this patent performs better than the official baseline provided by the MER2023 dataset in the emotion recognition task, and also surpasses the mainstream large visual models MAE and VideoMAE. This proves that expMAE can more effectively extract emotion-related information from visual modalities and convert it into effective feature representations. Compared with MAE and VideoMAE, expMAE can better capture subtle changes and semantic meanings of emotional information. This is attributed to expMAE's more refined and accurate perception of emotion-related static and dynamic features.

[0090] Table 2 Results of Single-Modal Emotion Recognition

[0091]

[0092] As shown in Table 3, the visual modalities extracted by expMAE proposed in this patented method perform well in emotion recognition tasks, not only in single-modal tasks but also in multimodal fusion. Because it incorporates both static and dynamic features, its fusion effect with other features is stronger than that of other visual modal features. Furthermore, the innovative inclusion of cross-modal features extracted by the CLIP model in this method effectively improves the accuracy of modality fusion.

[0093] Table 3. Results of Multimodal Fusion Emotion Recognition

[0094]

[0095] As shown in Table 4, the FBP model used in this patent method has achieved excellent results in comparison with the feature fusion schemes of traditional machine learning methods SVM and mainstream attention mechanisms.

[0096] Table 4 Comparison of Multimodal Fusion Models

[0097]

[0098] Finally, this method employs a weak-reinforcement semi-supervised learning strategy. From the unlabeled videos, some videos with distributions similar to the classes "worried" and "happy" are selected, given pseudo-labels, and added to the training set for further training. The final results are shown in Table 5, achieving an excellent F1 score of 88.55% on MER-SEMI, a 1.8% improvement over the official baseline of the MER2023 dataset.

[0099] Table 5 Comparison of Results for Semi-Supervised Learning Strategies

[0100]

[0101] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0102] In one embodiment, a multimodal emotion recognition device is provided, which corresponds one-to-one with the multimodal emotion recognition methods described in the above embodiments. For example... Figure 10 As shown, the multimodal emotion recognition device includes a labeled feature extraction unit 100, a model training unit 200, an unlabeled feature extraction unit 300, an unlabeled feature selection unit 400, and a fusion recognition unit 500. Detailed descriptions of each functional module are as follows:

[0103] The labeled feature extraction unit 100 is used to extract information of different labeled modalities from labeled video data, and to process the information of the different labeled modalities through a corresponding pre-trained model to extract labeled features.

[0104] The model training unit 200 is used to train the feature fusion model by using the labeled features as input to obtain a trained feature fusion model.

[0105] The unlabeled feature extraction unit 300 is used to extract information on different unlabeled modalities from unlabeled video data, and to process the information on the different unlabeled modalities through a corresponding pre-trained model to extract unlabeled features.

[0106] The unlabeled feature selection unit 400 is used to process the unlabeled features through a pre-trained feature fusion model to generate pseudo-labels, select features of pseudo-labels with higher confidence from the pseudo-labels, and add the features of pseudo-labels with higher confidence and the corresponding pseudo-labels to the training set.

[0107] The fusion recognition unit 500 is used to process the labeled data and selected pseudo-label data through a pre-trained feature fusion model to obtain the final recognition result and output it.

[0108] For specific limitations regarding multimodal emotion recognition devices, please refer to the limitations of multimodal emotion recognition methods described above, which will not be repeated here. Each module in the aforementioned multimodal emotion recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0109] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes a readable storage medium and internal memory. The readable storage medium stores an operating system, computer-readable instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The database stores data related to the multimodal emotion recognition method. The network interface communicates with external terminals via a network connection. When the computer-readable instructions are executed by the processor, a multimodal emotion recognition method is implemented. The readable storage medium provided in this embodiment includes both non-volatile and volatile readable storage media.

[0110] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 12 As shown, the computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor provides computational and control capabilities. The memory includes a readable storage medium and internal memory. The non-volatile storage medium stores the operating system and computer-readable instructions. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The network interface is used to communicate with an external server via a network connection. When the computer-readable instructions are executed by the processor, they implement a multimodal emotion recognition method. The readable storage medium provided in this embodiment includes both non-volatile and volatile readable storage media.

[0111] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor performs the following steps when executing the computer-readable instructions:

[0112] Information on different labeled modalities is extracted from labeled video data, and the information on the different labeled modalities is processed by a corresponding pre-trained model to extract labeled features.

[0113] The labeled features are used as input to the feature fusion model, and the feature fusion model is trained to obtain a trained feature fusion model.

[0114] Information on different unlabeled modalities is extracted from unlabeled video data, and the information on these different unlabeled modalities is processed by a corresponding pre-trained model to extract unlabeled features.

[0115] The unlabeled features are processed by the trained feature fusion model to generate pseudo-labels. Features with higher confidence are selected from the pseudo-labels and added to the training set along with the features with higher confidence and the corresponding pseudo-labels.

[0116] The labeled data and selected pseudo-label data are processed by a pre-trained feature fusion model to obtain the final recognition result, which is then output.

[0117] In one embodiment, one or more computer-readable storage media storing computer-readable instructions are provided. The readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage media. The readable storage media stores computer-readable instructions, which, when executed by one or more processors, perform the following steps:

[0118] Information on different labeled modalities is extracted from labeled video data, and the information on the different labeled modalities is processed by a corresponding pre-trained model to extract labeled features.

[0119] The labeled features are used as input to the feature fusion model, and the feature fusion model is trained to obtain a trained feature fusion model.

[0120] Information on different unlabeled modalities is extracted from unlabeled video data, and the information on these different unlabeled modalities is processed by a corresponding pre-trained model to extract unlabeled features.

[0121] The unlabeled features are processed by the trained feature fusion model to generate pseudo-labels. Features with higher confidence are selected from the pseudo-labels and added to the training set along with the features with higher confidence and the corresponding pseudo-labels.

[0122] The labeled data and selected pseudo-label data are processed by a pre-trained feature fusion model to obtain the final recognition result, which is then output.

[0123] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0124] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0125] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A multimodal emotion recognition method, characterized in that, The method includes the following steps: Information on different labeled modalities is extracted from labeled video data, and the information on the different labeled modalities is processed by a corresponding pre-trained model to extract labeled features. Using the labeled features as input to the feature fusion model, the feature fusion model is trained to obtain a trained feature fusion model. Extract information about different unlabeled modalities from unlabeled video data, and process the information about different unlabeled modalities using a corresponding pre-trained model to extract unlabeled features; The unlabeled features are processed by the trained feature fusion model to generate pseudo-labels. Features with higher confidence are selected from the pseudo-labels and added to the training set along with the features with higher confidence and the corresponding pseudo-labels. The labeled data and selected pseudo-label data are processed by the trained feature fusion model to obtain the final recognition result, which is then output. The information of different modalities includes image frames. The step of extracting labeled information of different modalities from labeled video data, or the step of extracting unlabeled information of different modalities from unlabeled video data, includes: A frame of the video is randomly selected as the input image for the current training round, and random masking and restoration operations are performed on the single image to capture the static features of facial expressions in each frame. Simultaneously, multiple video frames are used as input, and random masking and restoration operations are performed to extract dynamic features of facial expressions in the video. The information of different modalities includes text and image frames. The step of extracting labeled information of different modalities from labeled video data includes: Cross-features between text and image frames are extracted using the CLIP multimodal model; The labeled data and selected pseudo-label data respectively include dynamic features, static features, text features, audio features, and cross-features of text and image frames. The step of processing the labeled data and selected pseudo-label data using a trained feature fusion model to obtain the final recognition result includes: The dynamic and static features are processed using a bilinear pooling model to obtain the fused visual features. The cross features and text features of the text and image frames are processed by a bilinear pooling model to obtain fused visual text features; The visual features, visual text features, and audio features are processed using an Attention model to obtain the final fused feature representation. Emotion classification is performed on the final feature representation through a linear layer to obtain the final recognition result; Among them, the bilinear pooling model interacts with features of different modalities by performing bilinear projection and decomposition on the input features and generates a fused feature representation. Bilinear pooling models capture the nonlinear relationships between different modes by multiplying the features of different modes. Bilinear pooling models decompose input features, mapping high-dimensional feature representations to a low-dimensional space.

2. The multimodal emotion recognition method as described in claim 1, characterized in that, In the step of simultaneously using multiple video frames as input and performing random masking and restoration operations to extract dynamic features of facial expressions in the video, the random masking operation specifically means that the masking regions between different frames remain consistent.

3. The multimodal emotion recognition method as described in claim 1, characterized in that, The information of the different modalities includes audio. The step of extracting labeled modal information from labeled video data and processing the labeled modal information using a corresponding pre-trained model to extract labeled features includes: Extract labeled audio from labeled video data and convert the labeled audio into labeled Mel spectrograms; The labeled Mel spectrograms are processed using the HuberT model to extract labeled audio features. The step of extracting unlabeled information of different modalities from unlabeled video data, and processing the information of the different unlabeled modalities through a corresponding pre-trained model to extract unlabeled features, includes: Extract unlabeled audio from unlabeled video data and convert the unlabeled audio into an unlabeled Mel spectrogram; The unlabeled Mel spectrograms were processed using the HuBERT model to extract unlabeled audio features.

4. The multimodal emotion recognition method as described in claim 1, characterized in that, The information of different modalities includes text. The step of extracting labeled information of different modalities from labeled video data and processing the labeled information of different modalities through a corresponding pre-trained model to extract labeled features includes: Extract tagged text from tagged video data; The labeled text is processed using the MacBERT model to extract labeled text features; The step of extracting unlabeled information of different modalities from unlabeled video data, and processing the information of the different unlabeled modalities through a corresponding pre-trained model to extract unlabeled features, includes: Extracting unlabeled text from unlabeled video data; The labeled text is processed using the MacBERT model to extract labeled text features.

5. A multimodal emotion recognition device, comprising: The labeled feature extraction unit is used to extract information on different labeled modalities from labeled video data, and to process the information on the different labeled modalities through a corresponding pre-trained model to extract labeled features. The model training unit is used to train the feature fusion model by taking the labeled features as input to obtain the trained feature fusion model. The unlabeled feature extraction unit is used to extract information on different unlabeled modalities from unlabeled video data, and to process the information on the different unlabeled modalities through a corresponding pre-trained model to extract unlabeled features. The unlabeled feature selection unit is used to process the unlabeled features through a trained feature fusion model to generate pseudo-labels, select features with higher confidence from the pseudo-labels, and add the features with higher confidence and the corresponding pseudo-labels to the training set. as well as The fusion recognition unit is used to process the labeled data and the selected pseudo-label data through a pre-trained feature fusion model to obtain the final recognition result and output it. The information of different modalities includes image frames. The step of extracting labeled information of different modalities from labeled video data, or the step of extracting unlabeled information of different modalities from unlabeled video data, includes: A frame of the video is randomly selected as the input image for the current training round, and random masking and restoration operations are performed on the single image to capture the static features of facial expressions in each frame. Simultaneously, multiple video frames are used as input, and random masking and restoration operations are performed to extract dynamic features of facial expressions in the video. The information of different modalities includes text and image frames. Extracting labeled information of different modalities from labeled video data includes: Cross-features between text and image frames are extracted using the CLIP multimodal model; The labeled data and selected pseudo-label data respectively include dynamic features, static features, text features, audio features, and cross-features of text and image frames. The labeled data and selected pseudo-label data are processed using a pre-trained feature fusion model to obtain the final recognition result, including: The dynamic and static features are processed using a bilinear pooling model to obtain the fused visual features. The cross features and text features of the text and image frames are processed by a bilinear pooling model to obtain fused visual text features; The visual features, visual text features, and audio features are processed using an Attention model to obtain the final fused feature representation. Emotion classification is performed on the final feature representation through a linear layer to obtain the final recognition result; Among them, the bilinear pooling model interacts with features of different modalities by performing bilinear projection and decomposition on the input features and generates a fused feature representation. Bilinear pooling models capture the nonlinear relationships between different modes by multiplying the features of different modes. Bilinear pooling models decompose input features, mapping high-dimensional feature representations to a low-dimensional space.

6. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor, when executing the computer-readable instructions, implements the multimodal emotion recognition method as described in any one of claims 1-4.

7. One or more readable storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the multimodal emotion recognition method as described in any one of claims 1-4.

Images