Multi-modal emotion recognition method and device based on attention mechanism under collaborative learning

Abstract

The invention discloses a multi-modal emotion recognition method based on an attention mechanism under collaborative learning, and the method comprises the steps: obtaining a sample and a corresponding emotion state, and generating a training set, a verification set and a test set based on the sample; constructing a collaborative learning multi-modal emotion recognition model; defining a loss function of the collaborative learning multi-modal emotion recognition model; training a collaborative learning multi-modal emotion recognition model based on the training set; and inputting a to-be-predicted sample into the trained collaborative learning multi-modal emotion recognition model, and outputting a corresponding predicted emotion state. According to the feature fusion method combining the multi-head self-attention mechanism and the improved efficient additive attention module, the cross-modal global dependency relationship and the local context consistency can be effectively captured, and therefore better emotion recognition precision is achieved.

Description

Technical Field

[0001] The present invention relates to technical fields such as artificial intelligence, affective computing, and multimodal data analysis, and specifically to a multimodal emotion recognition method and device based on an attention mechanism in collaborative learning. By fusing cross-modal features of text and facial image data, high-precision emotion state analysis and classification in collaborative learning scenarios are achieved, thereby improving the dynamic monitoring and intervention capabilities of student interaction quality in educational scenarios. Background Art

[0002] Emotion recognition in collaborative learning is a core research area at the intersection of educational technology and artificial intelligence. By analyzing students' emotional states during collaborative learning, it supports optimizing teaching strategies and improving learning outcomes. In collaborative learning environments, emotion recognition can help teachers understand students' learning status, promote group communication and collaboration, and adjust teaching strategies through real-time emotion monitoring to optimize the learning experience. However, existing commonly used public multimodal emotion recognition datasets (such as IEMOCAP and CUM-MOSEI) suffer from a single emotion category and scenarios that are detached from actual collaborative learning environments. This limits the generalization ability of trained models in real-world educational scenarios.

[0003] Traditional emotion recognition methods rely on single-modal data (such as text or facial expressions). For example, Baziotis et al. proposed a Bi-LSTM model combined with a multi-layer self-attention mechanism to capture complex text patterns. However, the Bi-LSTM's performance is limited when handling long-range dependencies due to the vanishing gradient problem, making it difficult to capture long-range semantic connections. Furthermore, single-modal methods struggle to capture the complexity of emotion. For example, the same textual content, "This issue requires further discussion," may reflect a positive attitude when presented with a chuckle, but may also indicate anxiety when presented with a frown. This can easily lead to misjudgments based on single-modal analysis. Therefore, researchers have attempted to improve emotion recognition accuracy by combining multiple modalities. For example, Lee et al. integrated text and facial images into a multimodal model, significantly improving accuracy and highlighting the value of combining data fusion with deep learning. However, in most cases, modal fusion can contain redundant and interfering information. The richer the fused information, the higher the accuracy of sentiment analysis. Recent sentiment analysis research has focused on using deep learning techniques and multimodal methods to improve the performance of emotion recognition tasks. After inputting multimodal data, image and text features are extracted to obtain the representation information of each modality. Then, a suitable multimodal data fusion algorithm is selected to fuse the information of different modalities to generate a multimodal joint representation. Summary of the Invention

[0004] In response to the above-mentioned problems existing in the prior art, the present invention proposes a multimodal emotion recognition method and device based on the attention mechanism under collaborative learning.

[0005] The above-mentioned purpose of the present invention is achieved through the following technical solutions:

[0006] The multimodal emotion recognition method based on attention mechanism under collaborative learning includes the following steps:

[0007] Step 1: Obtain samples and corresponding emotional states, and generate training sets, validation sets, and test sets based on the samples. The samples include facial images and corresponding text inputs, and the text inputs include token IDs, attention masks, and paragraph identification results.

[0008] Step 2: Construct a collaborative learning multimodal emotion recognition model. The collaborative learning multimodal emotion recognition model includes a feature extractor, a feature fusion module, and a classifier. The sample is extracted with features by the feature extractor, then fused by the feature fusion module, and finally classified by the classifier. The feature fusion module includes a multi-head self-attention module and an additive attention module.

[0009] Step 3: Define the loss function of the collaborative learning multimodal emotion recognition model;

[0010] Step 4: Training the collaborative learning multimodal emotion recognition model based on the training set;

[0011] Step 5: Input the sample to be predicted into the trained collaborative learning multimodal emotion recognition model and output the corresponding predicted emotional state.

[0012] As described above, the emotional states in step 1 include positive leadership, positive dependence, positive relaxation, positive indifference, negative hostility, negative anxiety, negative contempt, negative indifference, and neutral.

[0013] As mentioned above, the sample in step 1 is obtained based on the following steps:

[0014] The text data of the participants' speeches and the personal facial video data during the speaking time are collected as original individual data. Each original individual data corresponds to an emotional state.

[0015] Personal images are extracted from personal facial video data at set time intervals, and personal images are denoised using a Gaussian filter. Face detection and cropping are then performed to obtain facial images. The resulting facial images are uniformly set to a pixel size and data enhancement is performed. Text data is segmented to obtain text input.

[0016] As mentioned above, the feature extractor in step 2 includes the BERT model and the VIT model. The BERT model performs Perform feature extraction to obtain text features , VIT model for facial images Extract features to obtain image features , align text features using a fully connected layer and image features The feature dimension of and features .

[0017] As mentioned above, the feature fusion module in step 2 includes multi-head self-attention and additive attention modules, and the splicing feature and features Get the initial fusion feature Fc:

[0018] ;

[0019] Get the weight matrix of the self-attention head, including the query matrix , key matrix , value matrix ,

[0020] ;

[0021] where · represents matrix multiplication, 、 、 are all learnable parameter matrices,

[0022] Compute the weighted output of the self-attention head :

[0023] ;

[0024] Among them, the feature dimension is the query matrix and bond matrix The number of columns, is a matrix The transpose of , Softmax() is the Softmax function,

[0025] The weighted outputs of each self-attention head are connected to finally obtain the fusion feature :

[0026] ;

[0027] in, is the output weight matrix, is the number of self-attention heads,

[0028] Get new query matrix and the new bond matrix :

[0029] ;

[0030] in, 、 is the learnable parameter matrix,

[0031] Calculating additive attention weights :

[0032] ;

[0033] in, is a learnable weight vector, is the scaling factor, represents scalar multiplication,

[0034] Calculate normalized attention weights :

[0035] ;

[0036] in, Represents the additive attention weight along Perform normalization operation,

[0037] Calculate the global context vector :

[0038] ;

[0039] Compute weighted attention output :

[0040] ;

[0041] in, As the projection function, Query vector The projection of the representation space,

[0042] Output the weighted attention As input, it is mapped to the category space through the fully connected layer, and the Softmax function is used to output the predicted classification probability.

[0043] As mentioned above, the loss function of the collaborative learning multimodal emotion recognition model is based on the following formula:

[0044] ;

[0045] in, Indicates the Is the true label of the sample Emotional state, Indicates the The samples are classified as The predicted classification probability of the class emotional state, represents the batch size, Represents the total number of emotional states.

[0046] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned identification method when executing the computer program.

[0047] A computer-readable storage medium stores a computer program, which implements the steps of the above-mentioned identification method when executed by a processor.

[0048] A computer program product includes a computer program, which implements the steps of the above identification method when executed by a processor.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. This paper proposes a feature fusion method that combines the multi-head self-attention mechanism (MHSA) and the improved efficient additive attention module (EAA), which can effectively capture cross-modal global dependencies and local context consistency, thereby achieving better emotion recognition accuracy.

[0051] 2. This paper constructs the emotional state of samples based on the psychological PAD model, including nine categories of emotional labels, covering real collaborative learning scenarios. It provides support for smart education applications, promotes the construction of smart classrooms, and lays a foundation for related research.

[0052] 3. The present invention adopts a two-stage fusion strategy, namely the multi-head self-attention mechanism MHSA models global interaction, and the efficient additive attention module EAA optimizes local alignment, which solves the representation inconsistency problem caused by the semantic gap between modalities in traditional methods.

[0053] 4. The present invention constructs samples and corresponding emotional states as labels, and uses the psychological PAD model to annotate the samples to enrich the emotional categories. Then, a series of data preprocessing methods are adopted. For text data, word segmentation and encoding operations are performed, attention masks are added, and [CLS] tags are added to unify the text length. For personal facial video data, image cropping is used to standardize the image size, and data enhancement technology is used to reduce overfitting and improve the generalization ability of the collaborative learning multimodal emotion recognition model. Finally, BERT is used to extract text features, the VIT model is used to extract image features, the multi-head self-attention mechanism and the improved additive attention module are used for feature fusion, and finally the Softmax function is used for classification. This achieves better emotion recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Flowchart of the present invention. DETAILED DESCRIPTION

[0055] In order to facilitate those skilled in the art to understand and implement the present invention, the present invention is further described in detail below in conjunction with embodiments. It should be understood that the embodiments described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0056] Example 1:

[0057] The multimodal emotion recognition method based on attention mechanism under collaborative learning includes the following steps:

[0058] Step 1: Design a collaborative learning experiment, build a collaborative learning emotion recognition dataset, and obtain training samples.

[0059] Step 1.1: Design the experimental scenario. The specific process includes: 1.1.1: Construct the discussion topic: Require the participants in the experimental group to clearly define the goals of the collaborative discussion topic. 1.1.2: Clarify basic concepts: Through pre-experimental training, ensure that the participants in the experimental group understand the background and concepts of the discussion topic. 1.1.3: Participants in the experimental group express their personal opinions: Require each participant to express their opinions independently. 1.1.4: Discuss opinions: Guide the participants in the experimental group to compare the similarities and differences of opinions and evaluate the pros and cons. Reflect on their own and others' positions. 1.1.5: Reach consensus: Require the participants in the experimental group to reach a unified conclusion.

[0060] Step 1.2: Multimodal data collection and annotation

[0061] The experiment recruited 12 students and divided them into six discussion groups. Each group was assigned a discussion topic (social issue) and required to complete collaborative tasks according to the process described in step 1.1. Data collection was conducted indoors using a built-in computer camera, an external HD camera, a microphone, and an online learning platform. The camera and microphone captured the participants' facial video and voice data. Text data was generated by converting the voice data. The text data of a student's speech, combined with the facial video data for that time, formed a raw individual data set. Emotional states were then categorized into nine types based on the three dimensions of the PAD model (pleasure P, arousal A, and dominance D). In this example, these included positive leadership; positive dependency; positive relaxation; positive indifference; negative hostility; negative anxiety; negative contempt; negative indifference; and neutral. Approximately 3,800 valid samples were obtained throughout the experiment, with each raw individual data set corresponding to a specific emotional state.

[0062] Step 1.3 Data Preprocessing

[0063] For personal facial video data, Python's OpenCV library is used to extract a frame of personal image from the personal facial video data at intervals of 6 seconds. The personal image is denoised using a Gaussian filter. Then, the Dlib library is used to detect and crop the face to obtain the facial image. Finally, the obtained facial image is standardized to 224 The size of the facial image is 224 pixels to meet the subsequent model input requirements. To enhance the generalization ability of the model and alleviate overfitting, the cropped facial image is subjected to random horizontal flipping (50% probability) for data augmentation to simulate the changes in facial posture under different shooting angles. The text data is generated by converting the original speech data. The preprocessing process for the text data includes using the WordPiece algorithm of the BERT model to segment the sentences of the text data, obtain subwords, add the [CLS] tag at the beginning of the input sentence, determine the token ID of each token of the subword, and to adapt to the batch input format, unify the length of the token ID and add an attention mask. In this embodiment, the BERT model is the MacBERT model.

[0064] Multimodal samples consist of facial images of students speaking and BERT-compatible text input, including token IDs, attention masks, and paragraph identification results. Each sample corresponds to an emotional state, with the corresponding emotional state serving as the ground-truth label. Sample pairs are randomly shuffled, and then training, validation, and test sets are generated in a 6:2:2 ratio.

[0065] Step 2: Build a collaborative learning multimodal emotion recognition model

[0066] The collaborative learning multimodal emotion recognition model established in this implementation includes feature extractors, feature fusion modules, classifiers, etc. Figure 1 As shown in Figure 2, the facial image and text input of the sample are extracted through the feature extractor, then fused through the feature fusion module, and finally classified using the classifier.

[0067] The feature extractor includes a BERT model and a VIT model. In this embodiment, the BERT model is a MacBERT model. The BERT model is used to extract the pre-processed text input. Perform feature extraction to obtain text features , VIT model for pre-processed facial images Extract features to obtain image features . The text features are then aligned using a fully connected layer and image features The feature dimension of and features . Subsequent features and features The fusion is performed through the feature fusion module, which includes the multi-head self-attention and additive attention modules. The entire feature fusion process can be described as:

[0068] Splicing features and features Get the initial fusion feature Fc:

[0069]

[0070] Get the weight matrix of the self-attention head, including the query matrix , key matrix , value matrix :

[0071]

[0072] where · represents matrix multiplication, 、 、 are all learnable parameter matrices.

[0073] Compute the weighted output of the self-attention head :

[0074]

[0075] Feature Dimension is the query matrix and bond matrix The number of columns, is a matrix , Softmax() is the Softmax function.

[0076] Connect the weighted outputs of each self-attention head to finally obtain the fusion feature :

[0077]

[0078] in, is the output weight matrix, is the number of self-attention heads.

[0079] The multi-head self-attention fusion described above models global cross-modal dependencies and captures the deep semantic interactions between images and text. The following explains the role of the additive attention module.

[0080] Get new query matrix and the new bond matrix :

[0081]

[0082] in, 、 is the learnable parameter matrix.

[0083] Calculating additive attention weights :

[0084]

[0085] in, is the additive attention weight, is a learnable weight vector, is a scaling factor based on the feature dimension, which is the new query matrix and the new bond matrix The reciprocal of the square root of the number of columns, Represents scalar multiplication.

[0086] Calculate normalized attention weights :

[0087]

[0088] The additive attention weights are normalized to become a probability distribution. represents normalization along the first dimension, Represents the additive attention weight along Perform a normalization (softmax) operation.

[0089] Obtain the global context vector through weighted summation :

[0090]

[0091] Using normalized attention weights For the new query matrix Perform weighted summation to obtain the global context vector , It is the weighted average of all new query vectors, reflecting global information.

[0092] Get weighted attention output :

[0093]

[0094] Using the global context vector For the new bond matrix Perform weighted operations to obtain new feature representations. Perform linear transformation on the above features, that is, through the projection function , mapping it from the original feature space to the query vector In the representation space, the projection function is a fully connected layer. The projection result is combined with the new query vector Add together to get the weighted attention output .

[0095] The additive attention module further refines and fuses features to enhance the consistency and discriminability of context representation. Finally, the weighted attention output As input, it is mapped to the category space through a fully connected layer and the Softmax function is used to output the predicted classification probability.

[0096] Step 3: Define the loss function for collaborative learning multimodal emotion recognition model

[0097] In this paper, the cross entropy loss function is used. The cross entropy loss function directly optimizes the discriminative ability of the collaborative learning multimodal emotion recognition model by measuring the difference between the predicted classification probability and the true label distribution. Its formula is:

[0098]

[0099] in, Indicates the Is the true label of the sample Class emotional state (1 if it is that class, 0 otherwise), Indicates the The samples are classified as The predicted classification probability of the class emotional state, Represents the batch size (Batch Size), Represents the total number of emotional states.

[0100] In classification tasks, the cross-entropy loss function uses the Softmax function to convert logits into probability distributions, directly modeling multi-classification tasks. By minimizing the negative log-likelihood, this loss function guides the collaborative learning multimodal emotion recognition model to focus on the most discriminative features for classification. Furthermore, the cross-entropy loss function's extractive properties make it highly effective during optimization: larger errors lead to stronger gradients, accelerating parameter adjustment for the collaborative learning multimodal emotion recognition model.

[0101] Step 4: Train the collaborative learning multimodal emotion recognition model based on the training set

[0102] To effectively train the collaborative learning multimodal emotion recognition model, the training set is used to train the collaborative learning multimodal emotion recognition model. Backpropagation is used to optimize parameters and learn the cross-modal associations between text and image features and the sentiment classification mapping relationship. The validation set is used to evaluate the performance of the collaborative learning multimodal emotion recognition model during training. It is used to adjust hyperparameters (such as learning rate and dropout rate) and monitor whether the collaborative learning multimodal emotion recognition model is overfitting, thereby improving its generalization ability. After the collaborative learning multimodal emotion recognition model is trained, the test set is used for final performance evaluation to verify the model's classification accuracy on unseen datasets.

[0103] In this example, the collaborative learning multimodal emotion recognition model was implemented using Pytorch, Python version 3.8. AdamW was used as the optimizer to prevent overfitting, and the learning rate was set to 3e-5. To prevent overfitting, the dropout function was applied for regularization. The RELU activation function enhances feature representation by introducing nonlinear transformations. The initial learning rates for the BERT model and the VIT model were set to 1e-5, and the batch size (BatchSize) was set to 16. The entire experimental process was conducted on an NVIDIA RTX A6000 GPU server, and each training session lasted 500 epochs.

[0104] Experimental results analysis

[0105] Comparative experiment

[0106] This embodiment will conduct comparative analysis on the following representative models and fusion methods.

[0107] (1) Resnet-50 model: The Resnet-50 model was proposed by Microsoft Research in 2015. It aims to capture deep features through residual connections and solve the problem of vanishing gradients in deep networks. It is an architecture widely used in computer vision tasks. In this study, Resnet-50 replaces the VIT model in this embodiment. This is to analyze the impact of different image feature extractors on the overall emotion recognition effect by controlling the text modality and fusion method unchanged.

[0108] (2) Chinese-MacBERT model: The Chinese-MacBERT model is an improved version of BERT. It adopts a more reasonable masking strategy, MLM as correction (Mac for short). In addition, it introduces mechanisms such as whole word masking (WWM), N-gram masking, and sentence order prediction (SOP). MacBERT performs well in multiple natural language processing tasks. The BERT model used in this article is MacBERT.

[0109] (3) Bert-base model: The Bert-base model is a Transformer model that is pre-trained in a self-supervised manner on a large amount of English data. It is designed to capture deep semantic representations in context and is widely used in natural language processing tasks. It has two pre-training objectives: masked language model (MLM) and next-round prediction (NSP). The general representation obtained after pre-training can be transferred to various downstream tasks. In this study, BERT-base replaces the MacBERT model in this embodiment. This move aims to analyze the impact of different text feature extractors on the overall sentiment recognition effect by controlling the image modality and fusion method unchanged.

[0110] (4) Roberta model: The Roberta model is also a Transformer-based model. It uses MLM as a pre-training target, removes the NSP task in BERT, and instead focuses on larger-scale MLM training, thereby improving training efficiency and model performance. In this study, Roberta replaces the MacBERT model in this embodiment. This move aims to analyze the impact of different text feature extractors on the overall sentiment recognition effect by controlling the image modality and fusion method unchanged.

[0111] (5) Chinese-bert-wwm-ext model: The Chinese-bert-wwm-ext model is a Chinese pre-trained model based on the BERT architecture. It adopts a whole-word masking (WWM) strategy, which enables the model to better understand word-level semantic information by masking entire words rather than individual subwords. In this study, Chinese-bert-wwm-ex replaces the MacBERT model in this embodiment. This is done to analyze the impact of different text feature extractors on the overall sentiment recognition effect while keeping the image modality and fusion method unchanged.

[0112] (6) NaiveCat model: The NaiveCat model is a simple feature fusion strategy that achieves fusion by splicing low-level and high-level features in the depth dimension. The spliced ​​feature map contains more contextual information. This method retains all input features, thereby achieving a richer feature representation. In this study, the NaiveCat method is used to replace the feature fusion method in this embodiment. This is to analyze the impact of different fusion methods on the overall emotion recognition effect under the premise of keeping the text modality and image modality unchanged.

[0113] (7) HSAEC model: The HSAEC model adopts a multi-branch fusion strategy. While retaining the independent branch features of each modality, it introduces an attention mechanism to promote interaction between modalities. Ultimately, the classification task is completed through multi-branch fusion. This method makes full use of single-modal information, enabling the model to integrate two levels of information, thereby improving the discriminative ability of the classifier. In this study, the HSAEC method is used to replace the feature fusion method in this embodiment. This is to analyze the impact of different fusion methods on the overall emotion recognition effect under the premise of keeping the text modality and image modality unchanged.

[0114] (8) CMAC model: Cross-modal Attention Fusion is a feature fusion strategy in multimodal learning. It uses the attention mechanism to weight features from different modalities, and achieves more flexible and refined feature fusion based on the contextual information of the input data and the interaction between modalities. This method can help the model better understand the relationship between different modalities, thereby improving the performance of multimodal tasks. In this study, the CMAC method is used to replace the feature fusion method in this embodiment. This is to analyze the impact of different fusion methods on the overall emotion recognition effect under the premise of keeping the text modality and image modality unchanged.

[0115] The comparative experimental results are as follows:

[0116] (1) The classification accuracy of the Resnet-50 model is 62.35%;

[0117] (2) The classification accuracy of this embodiment is 67.2%;

[0118] (3) The classification accuracy of the Bert-base model was 55.00%;

[0119] (4) The classification accuracy of the Roberta model was 57.50%;

[0120] (5) The classification accuracy of the Chinese-bert-wwm-ext model is 64.70%;

[0121] (6) The classification accuracy of the NaiveCat model is 41.47%;

[0122] (7) The classification accuracy of the HSAEC model was 58.97%;

[0123] (8) The classification accuracy using the CMAC model is 59.40%.

[0124] The comparison results with the above baseline model verify the effectiveness of the model proposed in this implementation.

[0125] Ablation experiments

[0126] This example designs an ablation study to verify the effectiveness of multimodal fusion. The experimental configuration is as follows: Experiment (1) Text-only: only text input is used for training and testing. Experiment (2) Img-only: only facial images are used for training and validation. Experiment (3) MHSA: facial images and text input are used for training and testing, but only multi-head self-attention is used for feature fusion. Experiment (4) ALL: the scheme described in Example 1 of the present invention. The experimental results are as follows:

[0127] The classification accuracy of experiment (1) was 62.7%;

[0128] The classification accuracy of experiment (2) was 55.2%;

[0129] The classification accuracy of experiment (3) was 63.5%;

[0130] The classification accuracy of experiment (4) is 67.2%.

[0131] The experimental results show that removing any information of facial images or text data will reduce the accuracy of sentiment analysis. The experimental results prove the effectiveness of the fusion method combining MHSA and additive attention module in the present invention.

[0132] Step 5: Input the sample to be predicted into the trained collaborative learning multimodal emotion recognition model and output the corresponding predicted emotional state.

[0133] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0134] Example 2:

[0135] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0136] Example 3:

[0137] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0138] Example 4:

[0139] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0140] The specific embodiments described herein are merely illustrative of the spirit of the present invention. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific embodiments without departing from the spirit of the present invention or exceeding the scope of the appended claims.

Claims

1. A multimodal emotion recognition method based on attention mechanism under collaborative learning, characterized by: The following steps are involved: Step 1: Obtain samples and corresponding emotional states, and generate training sets, validation sets, and test sets based on the samples. The samples include facial images and corresponding text inputs, and the text inputs include token IDs, attention masks, and paragraph identification results. Step 2: Construct a collaborative learning multimodal emotion recognition model. The collaborative learning multimodal emotion recognition model includes a feature extractor, a feature fusion module, and a classifier. The sample is extracted with features by the feature extractor, then fused by the feature fusion module, and finally classified by the classifier. The feature fusion module includes a multi-head self-attention module and an additive attention module. Step 3: Define the loss function of the collaborative learning multimodal emotion recognition model; Step 4: Training the collaborative learning multimodal emotion recognition model based on the training set; Step 5: Input the sample to be predicted into the trained collaborative learning multimodal emotion recognition model and output the corresponding predicted emotional state.

2. The multimodal emotion recognition method based on attention mechanism under collaborative learning according to claim 1 is characterized in that The emotional states in step 1 include positive leadership, positive dependence, positive relaxation, positive indifference, negative hostility, negative anxiety, negative contempt, negative indifference, and neutral.

3. The multimodal emotion recognition method based on attention mechanism under collaborative learning according to claim 1 is characterized in that The sample in step 1 is obtained based on the following steps: The text data of the participants' speeches and the personal facial video data during the speaking time are collected as original individual data. Each original individual data corresponds to an emotional state. Personal images are extracted from personal facial video data at set time intervals, and personal images are denoised using a Gaussian filter. Face detection and cropping are then performed to obtain facial images. The resulting facial images are uniformly set to a pixel size and data enhancement is performed. Text data is segmented to obtain text input.

4. The multimodal emotion recognition method based on attention mechanism under collaborative learning according to claim 1 is characterized in that The feature extractor in step 2 includes a BERT model and a VIT model. The BERT model performs text input Perform feature extraction to obtain text features , VIT model for facial images Extract features to obtain image features , align text features using a fully connected layer and image features The feature dimension of and features .

5. The multimodal emotion recognition method based on attention mechanism under collaborative learning according to claim 4 is characterized in that: The feature fusion module in step 2 includes multi-head self-attention and additive attention modules, and the splicing feature and features Get the initial fusion feature Fc: ； Get the weight matrix of the self-attention head, including the query matrix , key matrix , value matrix , ； where · represents matrix multiplication, 、 、 are all learnable parameter matrices, Compute the weighted output of the self-attention head : ； Among them, the feature dimension is the query matrix and bond matrix The number of columns, is a matrix The transpose of , Softmax() is the Softmax function, The weighted outputs of each self-attention head are connected to finally obtain the fusion feature : ； in, is the output weight matrix, is the number of self-attention heads, Get new query matrix and the new bond matrix : ； in, 、 is the learnable parameter matrix, Calculating additive attention weights : ； in, is a learnable weight vector, is the scaling factor, represents scalar multiplication, Calculate normalized attention weights : ； in, Represents the additive attention weight along Perform normalization operation, Calculate the global context vector : ； Compute weighted attention output : ； in, As the projection function, Query vector The projection of the representation space, Output the weighted attention As input, it is mapped to the category space through the fully connected layer, and the Softmax function is used to output the predicted classification probability.

6. The multimodal emotion recognition method based on attention mechanism under collaborative learning according to claim 5 is characterized in that: The loss function of the collaborative learning multimodal emotion recognition model is based on the following formula: ； in, Indicates the Is the true label of the sample Emotional state, Indicates the The samples are classified as The predicted classification probability of the class emotional state, represents the batch size, Represents the total number of emotional states.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the identification method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the identification method according to any one of claims 1 to 6 are implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the identification method according to any one of claims 1 to 6 are implemented.

Images