Multi-modal emotion recognition method and system based on pre-training model and multi-task learning

Abstract

The invention provides a multi-modal emotion recognition method based on a pre-training model and multi-task learning, and belongs to the technical field of deep learning application. The method specifically comprises the steps of collecting multi-modal data, and preprocessing original data; based on a self-supervised pre-training model, extracting universal representation and local abstract features by adopting a mode of combining different pre-training models with a CNN for different modal data, and constructing a mixed feature extraction network; on the basis of a cross attention Transformer, mining complementary emotion information of texts and voices through cross attention, further enhancing the ability to focus on key features through self-attention Bi-GRU, and constructing a network based on dual attention fusion; a multi-task learning strategy is introduced based on a feature extraction and feature fusion network, an auxiliary task is designed to help the model to better process cross-modal information, more useful features are captured, and a multi-modal emotion recognition model is obtained; the problems that in the prior art, emotion recognition classification is low in accuracy and poor in robustness are solved.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning application technology, and specifically relates to a multimodal emotion recognition method and system based on a pre-training model and multi-task learning. Background Art

[0002] With the rapid development of virtual worlds and the continuous advancement of artificial intelligence (AI), enhancing the human-computer interaction experience is becoming increasingly important and challenging. Emotion recognition technology enables computers to accurately understand human emotional states and provide intelligent responses to meet human needs. To achieve more natural human-computer interaction and a higher sense of immersion, emotion recognition has become a key component in improving human-computer interaction. Consequently, emotion recognition technology has become a key research direction in the field of AI, attracting widespread attention for its potential to revolutionize the way humans and computers interact. Emotion recognition is a fundamental task in natural language processing, computer vision, and speech processing, aiming to identify an individual's emotional state from multiple sources of information. However, traditional emotion recognition methods often focus on single-modality emotion analysis, ignoring the multidimensional nature of emotional expression. In reality, emotions are often generated by the interaction of multiple modalities (such as text, speech, and video). Single-modality analysis methods often exhibit poor robustness and accuracy when dealing with complex emotional changes or when information is missing. Traditional emotion recognition methods often rely on unimodal data. For example, text-based sentiment analysis relies on natural language processing techniques to capture emotional tendencies by analyzing vocabulary and syntax. Speech emotion recognition relies on acoustic features such as pitch, velocity, and energy. However, unimodal methods are susceptible to noise and information loss, resulting in performance limitations. For example, text modality may struggle to capture contextual emotional information, and speech modality may be affected by background noise.

[0003] With the rapid development of deep learning methods, researchers have realized that integrating multimodal information can significantly improve emotion recognition. In recent years, the rise of pre-trained and generative models, as well as the development of attention mechanisms, have provided strong technical support for multimodal emotion recognition. Existing methods have made significant progress in both theory and application, but several key issues remain that require further investigation. First, the ability to extract features from a single modality is limited, making it difficult to capture subtle differences in emotion. Second, cross-modal fusion mechanisms need to be refined, and the synergistic effects of multimodal information are not fully exploited.

[0004] With continuous breakthroughs in deep learning and cross-modal technologies, emotion recognition technology is showing broad application prospects. This technology not only enables basic emotion classification but also extends to more sophisticated tasks such as emotion intensity quantification and personalized feature modeling, significantly improving the intelligence and accuracy of emotion analysis. This multi-dimensional, personalized emotion analysis capability will have far-reaching application value in key areas such as healthcare and intelligent education. Summary of the Invention

[0005] The purpose of the present invention is to overcome the problems of low accuracy and weak robustness of multimodal emotion recognition in existing related technologies, and to provide a multimodal emotion recognition method and system based on pre-training models and multi-task learning.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a multimodal emotion recognition method based on a pre-trained model and multi-task learning, comprising the following steps: Collect multimodal data and preprocess the collected multimodal data; Based on the self-supervised pre-training model, different pre-training models are combined with CNN for different modal data to extract general representations and local abstract features, and build a hybrid feature extraction network; A cross-modal interaction network is constructed based on the Transformer with cross attention. The interdependence between different modalities is captured based on the cross-modal interaction network. The context dependency is captured through the self-attention Bi-GRU, and a feature fusion network based on dual attention is constructed. Based on the constructed hybrid feature extraction network and feature fusion network, a multi-task learning strategy is introduced to process cross-modal information by designing auxiliary tasks to build a multimodal emotion recognition model; Emotion recognition classification is performed based on the constructed multimodal emotion recognition model to obtain the emotion recognition results.

[0007] In the step of collecting multimodal data and preprocessing the collected multimodal data, the collected multimodal data includes text data and voice data.

[0008] In the step of constructing a hybrid feature extraction network based on a self-supervised pre-training model, different pre-training models are combined with CNN for different modal data to extract general representations and local abstract features. The hybrid feature extraction network consists of two channels, text and speech. RoBERTa is used for the text modality and Data2Vec is used for the speech modality.

[0009] The cross-attention-based Transformer constructs a cross-modal interaction network, captures the mutual dependencies between different modalities based on the cross-modal interaction network, captures context dependencies through self-attention Bi-GRU, and constructs a feature fusion network based on dual attention. In the step, the cross-modal interaction network is constructed by two cross-attention-based Transformers stacked in 3 layers.

[0010] The cross-modal interaction network captures the relationship between the two modalities by mining the complementary sentiment information of text and speech through cross-attention, using queries from one modality and keys and values ​​generated from another modality to capture the interdependencies between different modalities.

[0011] The cross-modal attention formula is as follows:

[0012]

[0013]

[0014]

[0015] in, is the text data modality query representation, and are the keys and values ​​of the voice data mode respectively, is the scaling factor; is the feature of text data modality; is the characteristic of the speech data modality; 、 and Represent weight values ​​respectively.

[0016] The multi-task learning strategy is introduced based on the constructed hybrid feature extraction network and feature fusion network, and cross-modal information is processed by designing auxiliary tasks to construct a multimodal emotion recognition model. The multi-task learning strategy includes a main task and an auxiliary task. The main task goal is a binary or seven-category emotion task, and the auxiliary task goal is to predict the intensity value of the emotion.

[0017] In the multi-task learning strategy, a loss function is introduced to measure the comprehensive performance of the multimodal emotion recognition model on multiple tasks and guide parameter updates. The main task loss function and the auxiliary task loss function are combined by weighted summation.

[0018] The main task loss function is the cross entropy loss function, and the specific formula is as follows:

[0019] in, For real emotional labels; Class probabilities predicted for multimodal emotion recognition models; The auxiliary task loss function adopts the mean square error loss function, and the specific formula is as follows:

[0020] in, Label for true emotional intensity; The sentiment intensity value predicted by the model; In a second aspect, the present invention provides a multimodal emotion recognition system based on a pre-trained model and multi-task learning, comprising: Data acquisition and processing module, used to collect multimodal data and pre-process the collected multimodal data; A hybrid feature extraction network construction module is used to extract general representations and local abstract features based on self-supervised pre-training models. It uses different pre-training models combined with CNN for different modal data to build a hybrid feature extraction network. The feature fusion network construction module is used to build a cross-modal interaction network based on the cross-attention Transformer, capture the interdependencies between different modalities based on the cross-modal interaction network, capture context dependencies through the self-attention Bi-GRU, and build a feature fusion network based on dual attention; A multimodal emotion recognition model construction module is used to construct a multimodal emotion recognition model by introducing a multi-task learning strategy based on the constructed hybrid feature extraction network and feature fusion network, and processing cross-modal information by designing auxiliary tasks; The recognition module is used to perform emotion recognition and classification based on the constructed multimodal emotion recognition model to obtain emotion recognition results.

[0021] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a multimodal emotion recognition method based on pre-training and multi-task learning, comprising the following steps: collecting multimodal data and preprocessing the collected multimodal data; Based on the self-supervised pre-training model, different pre-training models are combined with CNN for different modal data to extract general representations and local abstract features, and construct a hybrid feature extraction network; based on the cross-attention Transformer, a cross-modal interaction network is constructed, and the mutual dependencies between different modalities are captured based on the cross-modal interaction network. The context dependencies are captured through self-attention Bi-GRU, and a feature fusion network based on dual attention is constructed; based on the constructed hybrid feature extraction network and feature fusion network, a multi-task learning strategy is introduced, and cross-modal information is processed by designing auxiliary tasks to construct a multimodal emotion recognition model; emotion recognition classification is performed based on the constructed multimodal emotion recognition model to obtain emotion recognition results. By combining the pre-trained model with CNN to capture emotional information, rich features are extracted from a given modality, and the interaction ability between modalities is enhanced by combining the cross-attention-based Transformer. The context dependency is captured based on the self-attention Bi-GRU to form a more comprehensive emotional expression, which improves the model's recognition of the dynamic evolution of emotions and context-sensitive expressions, and greatly enhances the accuracy of model recognition. By introducing a multi-task learning strategy and designing auxiliary tasks to process cross-modal information, more useful features are captured, allowing the model to understand emotional expressions from multiple perspectives, improving the accuracy of emotional classification, and further improving the generalization and robustness of the model.

[0022] Furthermore, we employed fine-tuned pre-trained models, RoBERTa and Data2Vec, to extract universal representations for text and speech modal data, respectively. CNN was then added to capture local features, enhancing the feature representation and adapting to the spatial characteristics of multimodal data. This combination of two components more comprehensively captures sentiment information, laying a solid foundation for subsequent fusion processes.

[0023] Furthermore, a multi-task learning strategy is introduced to optimize the alignment mechanism between different modalities through the design of main tasks and auxiliary tasks, and to improve the recognition performance of the model through information sharing and collaborative training between different tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Flow chart of the method of the present invention; Figure 2 Schematic diagram of the network structure based on hybrid feature extraction in the present invention; Figure 3 Schematic diagram of the dual attention fusion network structure in the present invention; Figure 4 This is a downstream framework diagram of multi-task learning in the present invention; Figure 5 This is a structural diagram of the multimodal emotion recognition model in the present invention; Figure 6Schematic diagram of the comparison results of the binary and seven-category accuracy, F1 value, mean absolute error, and Pearson correlation value of different methods in the present invention on the CMU-MOSI dataset; Figure 7 Schematic diagram of the comparison results of the binary and seven-category accuracy, F1 value, mean absolute error, and Pearson correlation value of different methods in the present invention on the CMU-MOSEI dataset; Figure 8 A schematic diagram showing the comparative results of the convolutional neural network's auxiliary effect on pre-trained model feature extraction on the CMU-MOSI dataset, and the impact of the dual-attention-based fusion network and multi-task learning on the model's emotion recognition performance; Figure 9 This is a schematic diagram showing the comparative results of the convolutional neural network's auxiliary effect on pre-trained model feature extraction on the CMU-MOSEI dataset, the dual-attention-based fusion network, and the impact of multi-task learning on the model's emotion recognition performance. DETAILED DESCRIPTION

[0025] In order to further understand the content of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the embodiments are only for explaining the present invention and are not intended to limit it.

[0026] Example 1 like Figure 1 As shown in FIG, a multimodal emotion recognition method based on a pre-trained model and multi-task learning includes the following steps: S1: Collect multimodal data and preprocess the collected multimodal data; S2: Based on the self-supervised pre-training model, different pre-training models are combined with CNN for different modal data to extract general representations and local abstract features, and build a hybrid feature extraction network; S3: Build a cross-modal interaction network based on the cross-attention Transformer, capture the interdependence between different modalities based on the cross-modal interaction network, capture contextual dependencies through self-attention Bi-GRU, and build a feature fusion network based on dual attention; S4: Based on the constructed hybrid feature extraction network and feature fusion network, a multi-task learning strategy is introduced to process cross-modal information by designing auxiliary tasks to build a multimodal emotion recognition model; S5: Emotion recognition and classification are performed based on the constructed multimodal emotion recognition model to obtain emotion recognition results.

[0027] Preferably, in an embodiment of the present disclosure, the multimodal data collected in S1 includes text data and voice data, and the process of preprocessing the collected multimodal data is: Perform text cleaning operations on the collected text data (remove irrelevant characters and normalize spaces) and check whether the audio data is damaged.

[0028] Preferably, in the disclosed embodiment, the hybrid feature extraction network in S2 is: It consists of two channels, text and speech. The text modality adopts the RoBERTa model, and the speech modality adopts the Data2Vec model. A convolutional neural network (CNN) is added after the pre-training model to further focus on local features and optimize feature representation.

[0029] Preferably, in the embodiment of the present disclosure, the dual-attention-based feature fusion network in S3 includes cross-attention and self-attention, and the cross-modal interaction network is constructed by two cross-attention-based Transformers stacked in three layers.

[0030] Among them, the cross-modal interaction network mines the complementary emotional information of text and speech through cross-attention, captures the relationship between the two modalities, uses queries from one modality, and uses keys and values ​​generated from another modality to capture the interdependence between different modalities, that is, to perform feature fusion between modalities; After feature fusion, the self-attention-based Bi-GRU is used to capture contextual dependencies, further enhance the representation ability of key features, generate more discriminative joint representations, and improve the robustness of the model.

[0031] Preferably, in an embodiment of the present disclosure, the multi-task learning strategy in S4 includes a main task and an auxiliary task, wherein the main task goal is a sentiment binary classification task or a seven-classification task, and the auxiliary task goal is to predict the intensity value of the sentiment; In the multi-task learning strategy, a loss function is introduced to measure the comprehensive performance of the multimodal emotion recognition model on multiple tasks and guide parameter updating. The main task loss function and the auxiliary task loss function are combined by weighted summation. The main task loss function is the cross entropy loss function, and the auxiliary task loss function adopts the mean square error loss function.

[0032] Example 2 The method of the present invention is compared with the existing model, as follows: The present invention provides a method for extracting deep features from raw text and speech data on the public datasets CMU-MOSI and CMU-MOSEI using self-supervised pre-training models RoBERTa and Data2Vec, respectively, and combining them with convolutional neural networks to capture local abstract features in sequence data. Based on a dual attention fusion network, a three-layer stacked cross-attention-based Transformer is used as the interaction layer, combined with a self-attention-based bidirectional gated recurrent unit to fully mine the complementary information of text and speech modalities. A multi-task learning strategy is introduced to optimize the alignment mechanism between different modalities, and the recognition performance of the model is improved through information sharing and collaborative training between different tasks. Compared with existing methods, better classification accuracy is achieved, including the following steps: A hybrid feature extraction network is constructed, using fine-tuned pre-trained models RoBERTa and Data2Vec to extract universal representations for text and speech modal data, respectively. A CNN is then added to capture local features, enhancing the feature representation and adapting to the spatial characteristics of multimodal data. The combination of these two components more comprehensively captures sentiment information, laying a solid foundation for subsequent fusion. By combining pre-trained models with convolutional neural networks, the model is able to more comprehensively model multimodal sentiment information, thereby achieving better results in recognition tasks. Construct a dual-attention fusion network, including a feature interaction layer and an important emotional feature extraction layer. The feature interaction layer uses text and speech data as input, and is composed of two Transformers based on cross-modal cross-attention stacked in three layers. It uses the complementary emotional information in text and speech data to achieve information interaction between the two modalities through a cross-modal cross-attention mechanism, capturing the correlation between the modalities. The important emotional feature extraction layer is implemented by Bi-GRU combined with the self-attention mechanism. Bi-GRU has a powerful context capture capability when processing sequence data. Self-attention can automatically identify the contribution of different modal features and suppress the interference of noise modalities. Through cross-modal interaction and refined feature processing, the complementary information of text and speech is fully utilized to improve the performance of the model in emotion recognition tasks. In this example, experiments were conducted based on the original text and speech data from the public datasets CMU-MOSI and CMU-MOSEI, where: (1) CMU-MOSI is a multimodal emotion recognition dataset launched by Carnegie Mellon University, specifically for analyzing emotions and opinions in videos. It contains video clips from YouTube, where the conversations in the videos are about various topics. It includes text (extracted from subtitles), vision (facial expressions and other visual information extracted from video frames), and audio (speech and audio features extracted from video soundtracks). It contains a total of 2199 video clips (including text, speech, and video content), with an average video length of about 30 seconds. The dataset provides emotional labels for each video, which range from sentiment polarity of opinions / comments to regression tasks of sentiment intensity.

[0033] (2) CMU-MOSEI is an extended version of CMU-MOSI, providing more annotations and more complex sentiment analysis tasks. It is a multimodal sentiment and emotion analysis dataset that covers more emotional dimensions, not only limited to sentiment polarity and sentiment intensity, but also includes fine-grained annotations of sentiment and emotion. It contains multimodal information such as text, vision, and audio, and has 23,453 annotated video clips. Each video clip is annotated with multiple emotion categories and can contain multiple emotion labels, and these labels are usually multi-label.

[0034] The hybrid feature extraction network in the present invention is as follows Figure 2 As shown in the figure, to address key challenges faced by traditional methods, such as high data annotation costs, insufficient contextual understanding, and poor task versatility, fine-tuned pre-trained models RoBERTa and Data2Vec are used as feature extractors for text and speech modalities. Based on the Transformer architecture, the pre-trained model utilizes a self-attention mechanism to capture contextual information, significantly improving language comprehension. Through self-supervised learning, universal representations are learned from unlabeled data, reducing reliance on labeled data. The model can be fine-tuned on a variety of tasks, demonstrating enhanced versatility.

[0035] Text pre-training feature extraction model - RoBERTa: In 2019, Facebook AI proposed an improved pre-training language model based on BERT - RoBERTa (Robustly Optimized BERT Pretraining Approach), which significantly improved the performance of natural language processing tasks by optimizing the pre-training strategy.

[0036] The main improvements of RoBERTa are: (1) Large training data: RoBERTa uses more data for training than BERT, including a larger corpus.

[0037] (2) Long training time: RoBERTa further optimizes model performance by extending the training time.

[0038] (3) Dynamic mask: BERT uses a static mask, while RoBERTa dynamically generates a mask for each input, increasing the diversity of the data.

[0039] (4) Removing the NSP task: The Next Sentence Prediction (NSP) task in BERT was removed to focus on masked language modeling.

[0040] (5) Larger batch size: RoBERTa uses a larger batch size to improve training efficiency.

[0041] Speech pre-training feature extraction model - Data2Vec: A general self-supervised learning framework proposed by Meta AI in 2022. Its core concept is to learn unified feature representations from different views of the same data. Specifically, the model learns to predict the "teacher model" representation of the same input data. Specifically, the model extracts features from one view of the input data (such as masked text or a cropped image) and then predicts the representation of another view of the same data. The teacher model generates the target representation by encoding the complete input data. The student model, in turn, predicts the teacher model's output using partial input data.

[0042] Advantages of Data2Vec: (1) Multimodal support: Data2Vec can be applied to a variety of data types such as text, speech, and images, and has greater versatility.

[0043] (2) Unified framework: Through the teacher-student model and self-supervised learning objectives, Data2Vec avoids the complexity of designing specific tasks for each modality.

[0044] (3) Superior performance: Data2Vec performs as well as or better than dedicated models on text, speech, and image tasks, and can be seamlessly applied to data of different modalities without modifying the model architecture or training objectives.

[0045] Local feature refinement model - CNN: Combined with the global representation capabilities of the pre-trained model, CNN further extracts local correlation and structural information to construct a feature extractor that takes into account both local and global sentiment information.

[0046] The main functions of CNN are: (1) Local feature enhancement: For the word / sentence vector sequences output by RoBERTa, CNN uses 1D convolution kernels to capture local word order patterns, compensating for the Transformer's lack of sensitivity to local fine-grained features. The frame-level features extracted by Data2Vec may contain redundancy or noise in the temporal dimension. CNN can extract key acoustic expressions, retain significant emotional signals, and suppress irrelevant noise.

[0047] (2) Bridge for feature fusion: The pooling layer reduces the feature dimension and maps heterogeneous features to a space of uniform dimension, facilitating subsequent feature fusion.

[0048] Next, the dual attention-based feature fusion network of the present invention is constructed, such as Figure 3 As shown in the figure, cross-attention finely aligns and fuses feature information across modalities to form a more comprehensive emotional expression. A fusion network is constructed based on text and speech modalities. To accurately capture the correlation between modalities and reduce information loss, the feature interaction layer uses two stacked Transformers based on a cross-attention mechanism for fusion. The fused features generated after interaction are passed through a Bi-GRU based on a self-attention mechanism to further strengthen the understanding of the information represented by the new features within the modality. Finally, a fully connected layer is used for concatenation to complete the emotion recognition task.

[0049] Cross-modal attention formula:

[0050]

[0051]

[0052]

[0053] in, The query representation for the text data modality is denoted by , and the keys and values ​​from the speech data modality are denoted by and , is the scaling factor, i.e. the dimension of the key vector; is the feature of text data modality; is the characteristic of the speech data modality; 、 and Represent weight values ​​respectively. Text features Used to construct queries for the text Transformer encoder , also used to build keys for the audio Transformer encoder Sum Cross-attention essentially projects the hidden state from one modality into the space of another modality.

[0054] Then, the multi-task learning of the present invention is introduced into the downstream model, such as Figure 4 As shown in the figure, domain-specific information from auxiliary tasks is used to improve generalization performance, achieving better generalization and performance. By designing an auxiliary task (emotion intensity regression prediction), the upstream model is helped to learn unimodal and fused features. Features from different tasks guide the model to capture richer information and learn more general representations, allowing the model to understand emotional expressions from multiple perspectives and improve the accuracy of sentiment classification.

[0055] Main task - sentiment classification The goal is a binary or multi-classification task (such as positive and negative emotions, or multiple emotion types), using the cross entropy loss function:

[0056] in, For real emotional labels; Predicted class probabilities for a multimodal emotion recognition model.

[0057] Auxiliary task - sentiment intensity regression The goal is to predict the intensity value of the emotion (usually a continuous value indicating the intensity of the emotion), using the mean square error loss function:

[0058] in, Label for true emotional intensity; The sentiment intensity value predicted by the model.

[0059] Total loss function The total loss function combines the losses of multiple tasks by weighted summation. The losses of different tasks usually require different weights to balance their contributions to the total loss, and are adjusted based on the importance of the task or the effect during model training. This is shown below:

[0060] is the weight of the classification task; is the weight of the regression task.

[0061] The sentiment classification task outputs the classification result through a fully connected layer: for binary classification tasks, the Sigmoid activation function is used to output 0 or 1; for the seven-class classification task, the Softmax activation function is used to output the probability distribution of each class. The sentiment intensity regression task inputs the data into another fully connected layer for regression prediction. The regression head outputs a continuous sentiment intensity value using linear activation.

[0062] Finally, a multimodal emotion recognition model based on pre-training model and multi-task learning is constructed, such as Figure 5 shown.

[0063] Depend on Figure 6 and Figure 7 It can be seen intuitively that the proposed method has improved the binary classification, seven-classification and F1 value in the emotion recognition classification task compared with the MMML model. Figure 8 and Figure 9 The results verify the effectiveness of hybrid feature extraction network, dual attention fusion network and multi-task learning.

[0064] Example 3 Data acquisition and processing module, used to collect multimodal data and pre-process the collected multimodal data; A hybrid feature extraction network construction module is used to extract general representations and local abstract features based on self-supervised pre-training models. It uses different pre-training models combined with CNN for different modal data to build a hybrid feature extraction network. The feature fusion network construction module is used to build a cross-modal interaction network based on the cross-attention Transformer, capture the interdependencies between different modalities based on the cross-modal interaction network, capture context dependencies through the self-attention Bi-GRU, and build a feature fusion network based on dual attention; A multimodal emotion recognition model construction module is used to construct a multimodal emotion recognition model by introducing a multi-task learning strategy based on the constructed hybrid feature extraction network and feature fusion network, and processing cross-modal information by designing auxiliary tasks; The recognition module is used to perform emotion recognition and classification based on the constructed multimodal emotion recognition model to obtain emotion recognition results.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A multimodal emotion recognition method based on pre-training model and multi-task learning, characterized in that: The steps include: Collect multimodal data and preprocess the collected multimodal data; Based on the self-supervised pre-training model, different pre-training models are combined with CNN for different modal data to extract general representations and local abstract features, and build a hybrid feature extraction network; A cross-modal interaction network is constructed based on the Transformer with cross attention. The interdependence between different modalities is captured based on the cross-modal interaction network. The context dependency is captured through the self-attention Bi-GRU, and a feature fusion network based on dual attention is constructed. Based on the constructed hybrid feature extraction network and feature fusion network, a multi-task learning strategy is introduced to process cross-modal information by designing auxiliary tasks to build a multimodal emotion recognition model; Emotion recognition classification is performed based on the constructed multimodal emotion recognition model to obtain the emotion recognition results.

2. The multimodal emotion recognition method based on pre-training model and multi-task learning according to claim 1, characterized in that: In the step of collecting multimodal data and preprocessing the collected multimodal data, the collected multimodal data includes text data and voice data.

3. The multimodal emotion recognition method based on pre-training model and multi-task learning according to claim 2, characterized in that: In the step of constructing a hybrid feature extraction network based on a self-supervised pre-training model, different pre-training models are combined with CNN for different modal data to extract general representations and local abstract features. The hybrid feature extraction network consists of two channels, text and speech. RoBERTa is used for the text modality and Data2Vec is used for the speech modality.

4. The multimodal emotion recognition method based on pre-training model and multi-task learning according to claim 2, characterized in that: The cross-attention-based Transformer constructs a cross-modal interaction network, captures the mutual dependencies between different modalities based on the cross-modal interaction network, captures context dependencies through self-attention Bi-GRU, and constructs a feature fusion network based on dual attention. In the step, the cross-modal interaction network is constructed by two cross-attention-based Transformers stacked in 3 layers.

5. The multimodal emotion recognition method based on pre-training model and multi-task learning according to claim 4 is characterized in that: The cross-modal interaction network captures the relationship between the two modalities by mining the complementary sentiment information of text and speech through cross-attention, using queries from one modality and keys and values ​​generated from another modality to capture the interdependencies between different modalities.

6. The multimodal emotion recognition method based on pre-training model and multi-task learning according to claim 5, characterized in that: The cross-modal attention formula is as follows: in, is the text data modality query representation, and are the keys and values ​​of the voice data mode respectively, is the scaling factor; is the feature of text data modality; is the characteristic of the speech data modality; 、 and Represent weight values ​​respectively.

7. The multimodal emotion recognition method based on pre-training model and multi-task learning according to claim 2, characterized in that: The multi-task learning strategy is introduced based on the constructed hybrid feature extraction network and feature fusion network, and cross-modal information is processed by designing auxiliary tasks to construct a multimodal emotion recognition model. The multi-task learning strategy includes a main task and an auxiliary task. The main task goal is a binary or seven-category emotion task, and the auxiliary task goal is to predict the intensity value of the emotion.

8. The multimodal emotion recognition method based on pre-training model and multi-task learning according to claim 7, characterized in that: In the multi-task learning strategy, a loss function is introduced to measure the comprehensive performance of the multimodal emotion recognition model on multiple tasks and guide parameter updates. The main task loss function and the auxiliary task loss function are combined by weighted summation.

9. The multimodal emotion recognition method based on pre-training model and multi-task learning according to claim 8, characterized in that: The main task loss function is the cross entropy loss function, and the specific formula is as follows: in, For real emotional labels; Predicted class probabilities for multimodal emotion recognition models; The auxiliary task loss function adopts the mean square error loss function, and the specific formula is as follows: in, Label for true emotional intensity; The sentiment intensity value predicted by the model.

10. A multimodal emotion recognition system based on a pre-training model and multi-task learning according to any one of claims 1 to 9, characterized in that: include: Data acquisition and processing module, used to collect multimodal data and pre-process the collected multimodal data; A hybrid feature extraction network construction module is used to extract general representations and local abstract features based on self-supervised pre-training models. It uses different pre-training models combined with CNN for different modal data to build a hybrid feature extraction network. The feature fusion network construction module is used to build a cross-modal interaction network based on the cross-attention Transformer, capture the interdependencies between different modalities based on the cross-modal interaction network, capture context dependencies through the self-attention Bi-GRU, and build a feature fusion network based on dual attention; A multimodal emotion recognition model construction module is used to construct a multimodal emotion recognition model by introducing a multi-task learning strategy based on the constructed hybrid feature extraction network and feature fusion network, and processing cross-modal information by designing auxiliary tasks; The recognition module is used to perform emotion recognition and classification based on the constructed multimodal emotion recognition model to obtain emotion recognition results.

Images