Collaborative learning based multi-modal sentiment recognition method and device based on attention mechanism

Abstract

This invention discloses a multimodal emotion recognition method based on an attention mechanism under collaborative learning. The method acquires samples and their corresponding emotional states, generates training, validation, and test sets based on the samples, constructs a collaborative learning multimodal emotion recognition model, defines the loss function of the model, trains the model on the training set, and inputs the sample to be predicted into the trained model to output the corresponding predicted emotional state. This invention proposes a feature fusion method combining a multi-head self-attention mechanism and an improved, efficient additive attention module, which can effectively capture cross-modal global dependencies and local contextual consistency, thereby achieving better emotion recognition accuracy.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, affective computing, and multimodal data analysis, specifically to a multimodal emotion recognition method and device based on attention mechanism in collaborative learning. By fusing cross-modal features of text and facial image data, it achieves high-precision emotion state analysis and classification in collaborative learning scenarios, thereby improving the dynamic monitoring and intervention capabilities for student interaction quality in educational settings. Background Technology

[0002] Emotion recognition in collaborative learning is a core research direction at the intersection of educational technology and artificial intelligence. By analyzing students' emotional states during collaborative learning, it provides support for optimizing teaching strategies and improving learning outcomes. In a collaborative learning environment, emotion recognition can help teachers understand students' learning status, promote group communication and collaboration, and adjust teaching strategies to optimize the learning experience by monitoring emotions in real time. However, existing commonly used public multimodal emotion recognition datasets (such as IEMOCAP and CUM-MOSEI) suffer from problems such as limited emotion categories and scenarios that are not closely related to real-world collaborative learning environments, resulting in limited generalization ability of trained models in real educational scenarios.

[0003] Traditional sentiment recognition methods rely on single-modal data (such as text or facial expressions). For example, Baziotis et al. proposed a Bi-LSTM model combining multi-layer self-attention mechanisms to capture complex text patterns. However, Bi-LSTM suffers from the vanishing gradient problem, limiting its performance in handling long-range dependencies and making it difficult to capture distant semantic associations. Furthermore, single-modal methods struggle to capture the complexity of emotions. For instance, the same text, "This issue needs further discussion," might reflect a positive attitude with a smile, but imply anxiety with a frown, making single-modal analysis prone to misjudgment. Therefore, researchers have attempted to improve sentiment recognition accuracy by combining multi-modal information. For example, Lee et al. integrated text and facial images into a multimodal model, significantly improving accuracy and highlighting the value of combining data fusion with deep learning. However, in most cases, modality fusion may contain redundant interfering information. Richer fused information does not necessarily lead to higher sentiment analysis accuracy. Recent sentiment analysis research focuses on using deep learning techniques and multimodal methods to improve the performance of sentiment recognition tasks. After inputting multimodal data, image and text features are extracted to obtain the representation information of each modality. Then, a suitable multimodal data fusion algorithm is selected to fuse the information of different modalities, thereby generating a multimodal joint representation. Summary of the Invention

[0004] To address the aforementioned problems in existing technologies, this invention proposes a multimodal emotion recognition method and device based on an attention mechanism under collaborative learning.

[0005] The above-mentioned objective of this invention is achieved through the following technical solution:

[0006] A multimodal emotion recognition method based on attention mechanism under collaborative learning includes the following steps:

[0007] Step 1: Obtain samples and corresponding emotional states. Generate training, validation, and test sets based on the samples. Samples include facial images and corresponding text inputs. Text inputs include Token IDs, attention masks, and paragraph labeling results.

[0008] Step 2: Construct a collaborative learning multimodal emotion recognition model. The collaborative learning multimodal emotion recognition model includes a feature extractor, a feature fusion module, and a classifier. The sample extracts features through the feature extractor, then fuses them through the feature fusion module, and finally classifies them using the classifier. The feature fusion module includes multi-head self-attention and additive attention modules.

[0009] Step 3: Define the loss function for the collaborative learning multimodal emotion recognition model;

[0010] Step 4: Train the collaborative learning multimodal emotion recognition model based on the training set;

[0011] Step 5: Input the sample to be predicted into the trained collaborative learning multimodal emotion recognition model and output the corresponding predicted emotion state.

[0012] As mentioned above, the emotional states in step 1 include positive leadership, positive dependence, positive relaxation, positive irrelevance, negative hostility, negative anxiety, negative contempt, negative irrelevance, and neutrality.

[0013] As described above, the sample in step 1 is obtained based on the following steps;

[0014] Text data of participants' speeches and personal facial video data during the speaking time were collected as raw individual data. Each raw individual data point corresponds to an emotional state.

[0015] Personal images are extracted from personal facial video data at set time intervals, and personal images are denoising through a Gaussian filter. Then, face detection and cropping are performed to obtain facial images. The obtained facial images are uniformly set in pixel size and data augmentation is performed. Text data is segmented to obtain text input.

[0016] As mentioned above, the feature extractors in step 2 include the BERT model and the VIT model. The BERT model extracts text input... Text features are obtained by feature extraction. VIT model for facial images Image features are obtained by extracting features. Use fully connected layers to align text features and image features Feature dimensions, to obtain features and characteristics .

[0017] As mentioned above, the feature fusion module in step 2 includes multi-head self-attention and additive attention modules, which concatenate features. and characteristics Preliminary fusion feature Fc is obtained:

[0018] ;

[0019] Obtain the weight matrix of the self-attention head, including the query matrix. Key matrix Value matrix ,

[0020] ;

[0021] Where · denotes matrix multiplication. , , All of these are learnable parameter matrices.

[0022] Calculate the weighted output of the self-attention head :

[0023] ;

[0024] Among them, feature dimension It is a query matrix Bond matrix The number of columns, It is a matrix The transpose of , Softmax() is the Softmax function,

[0025] The weighted outputs of each self-attention point are concatenated to obtain the fused feature. :

[0026] ;

[0027] in, It is the output weight matrix. It is the number of self-focused heads.

[0028] Get the new query matrix and new key matrix :

[0029] ;

[0030] in, , For a learnable parameter matrix,

[0031] Calculate additive attention weights :

[0032] ;

[0033] in, It is a learnable weight vector. It is a scaling factor. This represents scalar multiplication.

[0034] Calculate normalized attention weights :

[0035] ;

[0036] in, Represents additive attention weights along Perform normalization operation.

[0037] Calculate the global context vector :

[0038] ;

[0039] Calculate weighted attention output :

[0040] ;

[0041] in, For the projection function, To query vector The representation space projection it is located in.

[0042] Weighted attention output As input, the data is mapped to the category space through a fully connected layer, and the predicted classification probability is output using the Softmax function.

[0043] As mentioned above, the loss function of the collaborative learning multimodal emotion recognition model is based on the following formula:

[0044] ;

[0045] in, Indicates the first Is the true label of the sample the first one? Emotional states, Indicates the first The sample was classified as the first... Predicted classification probability of emotion states Represents batch size, This represents the total number of emotional states.

[0046] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the above-described identification method.

[0047] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described identification method.

[0048] A computer program product includes a computer program that, when executed by a processor, implements the steps of the above-described identification method.

[0049] Compared with the prior art, the present invention has the following advantages:

[0050] 1. This invention proposes a feature fusion method that combines multi-head self-attention mechanism (MHSA) and improved efficient additive attention module (EAA), which can effectively capture cross-modal global dependencies and local contextual consistency, thereby achieving better emotion recognition accuracy.

[0051] 2. This invention constructs emotional states of samples based on the psychological PAD model, including 9 categories of emotional labels, covering real-world collaborative learning scenarios. It provides support for smart education applications, promotes the construction of smart classrooms, and lays the foundation for related research.

[0052] 3. This invention solves the representation inconsistency problem caused by the semantic gap between modalities in traditional methods by adopting a two-stage fusion strategy, namely, multi-head self-attention mechanism (MHSA) for global interaction modeling and efficient additive attention module (EAA) for local alignment optimization.

[0053] 4. This invention constructs samples and corresponding emotional states as labels, and uses a psychological PAD model to annotate the samples to enrich the emotional categories. Then, a series of data preprocessing methods are employed. For text data, word segmentation and encoding operations are performed, attention masks are added, and [CLS] tags are added to standardize text length. For personal facial video data, image cropping is used to standardize image size, and data augmentation techniques are used to mitigate overfitting and improve the generalization ability of the collaborative learning multimodal emotion recognition model. Finally, BERT is used to extract text features, the VIT model is used to extract image features, a multi-head self-attention mechanism and an improved additive attention module are used for feature fusion, and the Softmax function is used for classification. This achieves better emotion recognition results. Attached Figure Description

[0054] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0055] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to embodiments. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0056] Example 1:

[0057] A multimodal emotion recognition method based on attention mechanism under collaborative learning includes the following steps:

[0058] Step 1: Design collaborative learning experiments, construct a collaborative learning emotion recognition dataset, and obtain training samples.

[0059] Step 1.1: Design the Experimental Scenario. The specific process includes: 1.1.1: Constructing the Discussion Topic: Participants in the experimental group are required to clarify the goals of the collaborative discussion topic. 1.1.2: Clarifying Basic Concepts: Through pre-experiment training, ensure participants understand the background and concepts of the discussion topic. 1.1.3: Participants in the Experimental Group Express Their Individual Viewpoints: Each participant is required to independently state their viewpoint. 1.1.4: Discussing Viewpoints: Guide participants to compare the similarities and differences in their viewpoints, evaluate their strengths and weaknesses, and reflect on their own and others' positions. 1.1.5: Reaching a Consensus: Participants in the experimental group are required to reach a unified conclusion.

[0060] Step 1.2: Multimodal Data Collection and Labeling

[0061] The experiment recruited 12 students, divided into 6 discussion groups. Each group was assigned a discussion topic (social issue) and was required to complete the collaborative task according to the process in step 1.1. During data collection, built-in computer cameras, external high-definition cameras, microphones, and online learning platform tools were used to collect data under indoor conditions. Cameras and microphones were used to capture participants' personal facial video and audio data. Text data was generated from the converted audio data. The text data of a student's (participant's) speech, combined with their facial video data during that time, constituted a raw individual data set. Subsequently, based on the three dimensions of the PAD model (pleasure (P), arousal (A), and dominance (D),) emotional states were categorized into 9 types: in this embodiment, these include positive leadership, positive dependence, positive relaxation, positive irrelevance, negative hostility, negative anxiety, negative contempt, negative irrelevance, and neutrality. The total number of valid samples obtained in the experiment was approximately 3800, with each raw individual data set corresponding to one emotional state.

[0062] Step 1.3 Data Preprocessing

[0063] For personal facial video data, the OpenCV library in Python was used to extract a frame of personal image from the video data at 6-second intervals. Noise reduction was performed on the personal image using a Gaussian filter. Then, the Dlib library was used for face detection and cropping to obtain facial images. Finally, the obtained facial images were standardized to 224. The image size is 224 pixels to meet the input requirements of subsequent models. To enhance the model's generalization ability and alleviate overfitting, the cropped facial image is randomly horizontally flipped (50% probability) for data augmentation to simulate facial pose changes under different shooting angles. The text data is generated from the original speech data. The preprocessing of the text data includes using the WordPiece algorithm of the BERT model to segment the sentences of the text data to obtain sub-words, adding a [CLS] marker at the beginning of the input sentence, determining the Token ID of each Token of the sub-words, and uniformizing the length of the Token IDs to adapt to the batch processing input format, and adding an attention mask. In this embodiment, the BERT model is the MacBERT model.

[0064] The multimodal samples include facial images of students speaking and BERT-compatible text input, which includes Token IDs, attention masks, and paragraph labels. Each sample corresponds to an emotional state, and the emotional state of the sample is the ground truth label. The sample pairs are randomly shuffled, and then training, validation, and test sets are generated in a 6:2:2 ratio.

[0065] Step 2: Establish a collaborative learning multimodal emotion recognition model

[0066] The collaborative learning multimodal emotion recognition model established in this implementation includes a feature extractor, a feature fusion module, and a classifier. For example... Figure 1 As shown, the facial image and text input of the sample are used to extract features through a feature extractor, then fused through a feature fusion module, and finally classified using a classifier.

[0067] Feature extractors include the BERT model and the VIT model. In this embodiment, the BERT model is the MacBERT model. The BERT model is used to extract the preprocessed text input. Text features are obtained by feature extraction. The VIT model processes preprocessed facial images. Image features are obtained by extracting features. Then, a fully connected layer is used to align text features. and image features The feature dimensions are used to ultimately obtain the features. and characteristics Subsequent features and characteristics Feature fusion is performed through a feature fusion module, which includes multi-head self-attention and additive attention modules. The entire feature fusion process can be described as follows:

[0068] splicing features and characteristics Preliminary fusion feature Fc is obtained:

[0069]

[0070] Obtain the weight matrix of the self-attention head, including the query matrix. Key matrix Value matrix :

[0071]

[0072] Where · denotes matrix multiplication. , , All of these are learnable parameter matrices.

[0073] Calculate the weighted output of the self-attention head :

[0074]

[0075] Feature Dimension It is a query matrix Bond matrix The number of columns, It is a matrix The transpose of , Softmax() is the Softmax function.

[0076] By concatenating the weighted outputs of each self-attention point, the fused feature is finally obtained. :

[0077]

[0078] in, It is the output weight matrix. It is the number of self-focused heads.

[0079] The multi-head self-attention fusion described above models cross-modal global dependencies, capturing deep semantic interactions between images and text. The role of the additive attention module is explained below.

[0080] Get the new query matrix and new key matrix :

[0081]

[0082] in, , is a learnable parameter matrix.

[0083] Calculate additive attention weights :

[0084]

[0085] in, It is additive attention weight. It is a learnable weight vector. It is a scaling factor based on the feature dimension; it is the new query matrix. and new key matrix The reciprocal of the square root of the sequence number. This indicates scalar multiplication.

[0086] Calculate normalized attention weights :

[0087]

[0088] The additive attention weights are normalized to make them a probability distribution. This indicates normalization along the first dimension. Represents additive attention weights along Perform a normalization (softmax) operation.

[0089] The global context vector is obtained by weighted summation. :

[0090]

[0091] Using normalized attention weights For the new query matrix Perform a weighted summation to obtain the global context vector. , It is a weighted average of all new query vectors, reflecting global information.

[0092] Obtain weighted attention output :

[0093]

[0094] Use global context vector For the new key matrix Weighted operations are performed to obtain new feature representations. A linear transformation is then applied to these features, i.e., through a projection function. Map it from the original feature space to the query vector In the representation space, the projection function is a fully connected layer. The projection result is then compared with the new query vector. Add them together to get the weighted attention output. .

[0095] The additive attention module further refines and fuses features, enhancing the consistency and discriminability of contextual representations. Finally, the weighted attention output is... As input, the data is mapped to the category space through a fully connected layer, and the predicted classification probability is output using the Softmax function.

[0096] Step 3: Define the loss function for the collaborative learning multimodal emotion recognition model.

[0097] In this invention, the cross-entropy loss function is employed. The cross-entropy loss function directly optimizes the class discrimination ability of the collaborative learning multimodal emotion recognition model by measuring the difference between the predicted classification probability and the true label distribution. Its formula is:

[0098]

[0099] in, Indicates the first Is the true label of the sample the first one? Emotional state category (1 if it is this category, 0 otherwise). Indicates the first The sample was classified as the first... Predicted classification probability of emotion states Represents the batch size. This represents the total number of emotional states.

[0100] In classification tasks, the cross-entropy loss function uses the Softmax function to transform logits into a probability distribution, directly modeling multi-class classification tasks. This loss function guides the collaborative learning multimodal sentiment recognition model to focus on the features most discriminative for classification by minimizing the negative log-likelihood. Furthermore, the extraction characteristics of the cross-entropy loss function make it perform well in the optimization process; the larger the error, the stronger the gradient, accelerating the parameter tuning of the collaborative learning multimodal sentiment recognition model.

[0101] Step 4: Train the collaborative learning multimodal emotion recognition model based on the training set.

[0102] To effectively train the collaborative learning multimodal emotion recognition model, a training set is used to train the model, optimizing parameters through backpropagation and learning cross-modal associations of text and image features, as well as emotion classification mappings. A validation set is used to evaluate the model's performance during training, for tuning hyperparameters (such as learning rate and Dropout rate), and to monitor for overfitting, thereby improving its generalization ability. A test set is used for final performance evaluation after training, validating the model's classification accuracy on unseen datasets.

[0103] In this embodiment, the collaborative learning multimodal emotion recognition model is implemented using PyTorch, Python version 3.8. AdamW is used as the optimizer to prevent overfitting, and the learning rate is set to 3e-5. Dropout is applied for regularization to prevent overfitting. The ReLU activation function enhances feature representation by introducing a non-linear transformation. The initial learning rate for both the BERT and VIT models is set to 1e-5, and the batch size is set to 16. The entire experiment was conducted on an NVIDIA RTX A6000 GPU server, with each training iteration lasting 500 epochs.

[0104] Analysis of Experimental Results

[0105] Comparative experiment

[0106] This embodiment will be compared and analyzed with the following representative models and fusion methods.

[0107] (1) ResNet-50 model: The ResNet-50 model was proposed by Microsoft Research in 2015. It aims to capture deep features through residual connections and solve the gradient vanishing problem in deep networks. It is an architecture widely used in computer vision tasks. In this study, ResNet-50 is used instead of the VIT model in this embodiment. This is to analyze the impact of different image feature extractors on the overall emotion recognition effect while keeping the text modality and fusion method unchanged.

[0108] (2) Chinese-MacBERT Model: The Chinese-MacBERT model is an improved version of BERT, which adopts a more reasonable masking strategy, MLM as correction (Mac). In addition, it introduces mechanisms such as whole-word masking (WWM), N-gram masking, and sentence order prediction (SOP). MacBERT performs well in multiple natural language processing tasks. The BERT model used in this paper is MacBERT.

[0109] (3) Bert-base model: The Bert-base model is a Transformer model pre-trained in a self-supervised manner on a large amount of English data. It is designed to capture deep semantic representations in context and is widely used in natural language processing tasks. It has two pre-training objectives: Masked Language Model (MLM) and Next-Game Prediction (NSP). The general representations obtained after pre-training can be transferred to various downstream tasks. In this study, BERT-base is used instead of the MacBERT model in this embodiment. This is to analyze the impact of different text feature extractors on the overall sentiment recognition performance while keeping the image modality and fusion method unchanged.

[0110] (4) Roberta Model: The Roberta model is also a Transformer-based model. It uses MLM as the pre-training target, removes the NSP task in BERT, and focuses on larger-scale MLM training, thereby improving training efficiency and model performance. In this study, Roberta is used instead of the MacBERT model in this embodiment. This is to analyze the impact of different text feature extractors on the overall sentiment recognition effect while keeping the image modality and fusion method unchanged.

[0111] (5) Chinese-bert-wwm-ext model: The Chinese-bert-wwm-ext model is a Chinese pre-trained model based on the BERT architecture. It adopts the whole-word masking (WWM) strategy, which masks the entire word rather than individual sub-words, enabling the model to better understand word-level semantic information. In this study, Chinese-bert-wwm-ex replaces the MacBERT model in this embodiment. This is to analyze the impact of different text feature extractors on the overall sentiment recognition effect while keeping the image modality and fusion method unchanged.

[0112] (6) NaiveCat Model: The NaiveCat model is a simple feature fusion strategy that achieves fusion by concatenating low-level and high-level features in the depth dimension. The concatenated feature map contains more contextual information. This method preserves all input features, thus achieving richer feature representation. In this study, the NaiveCat method is used to replace the feature fusion method in this embodiment. This is intended to analyze the impact of different fusion methods on the overall sentiment recognition performance while keeping the text modality and image modality unchanged.

[0113] (7) HSAEC Model: The HSAEC model employs a multi-branch fusion strategy, retaining the independent branch features of each modality while introducing an attention mechanism to promote inter-modal interaction. Ultimately, the classification task is completed through multi-branch fusion. This method fully utilizes single-modal information, enabling the model to integrate information from two levels, thereby improving the classifier's discriminative ability. In this study, the HSAEC method is used to replace the feature fusion method in this embodiment. This aims to analyze the impact of different fusion methods on the overall emotion recognition effect while keeping the text modality and image modality unchanged.

[0114] (8) CMAC Model: Cross-modal Attention Fusion (CMAC) is a feature fusion strategy in multimodal learning. It utilizes an attention mechanism to weight features from different modalities, achieving more flexible and refined feature fusion based on the contextual information of the input data and the interactions between modalities. This method helps the model better understand the relationships between different modalities, thereby improving the performance of multimodal tasks. In this study, the CMAC method is used to replace the feature fusion method in this embodiment. This is intended to analyze the impact of different fusion methods on the overall emotion recognition effect while keeping the text modality and image modality unchanged.

[0115] The results of the comparative experiment are as follows:

[0116] (1) The classification accuracy using the ResNet-50 model was 62.35%;

[0117] (2) The classification accuracy rate of this embodiment is 67.2%;

[0118] (3) The classification accuracy using the Bert-base model was 55.00%;

[0119] (4) The classification accuracy using the Roberta model was 57.50%;

[0120] (5) The classification accuracy using the Chinese-bert-wwm-ext model was 64.70%;

[0121] (6) The classification accuracy using the NaiveCat model was 41.47%;

[0122] (7) The classification accuracy using the HSAEC model was 58.97%;

[0123] (8) The classification accuracy using the CMAC model was 59.40%.

[0124] The comparison results with the baseline model above verify the effectiveness of the model proposed in this implementation.

[0125] ablation experiment

[0126] This embodiment designs an ablation study to verify the effectiveness of multimodal fusion. The experimental configuration is described below: Experiment (1) Text-only: Training and testing are performed using only text input. Experiment (2) Img-only: Training and verification are performed using only facial images. Experiment (3) MHSA: Training and testing are performed using both facial images and text input, but feature fusion is performed using only multi-head self-attention. Experiment (4) ALL: The scheme described in Embodiment 1 of this invention. The experimental results are as follows:

[0127] The classification accuracy of experiment (1) was 62.7%;

[0128] The classification accuracy of experiment (2) was 55.2%;

[0129] The classification accuracy of experiment (3) was 63.5%;

[0130] The classification accuracy of Experiment (4) was 67.2%.

[0131] Experimental results show that removing any information from facial images or text data reduces the accuracy of sentiment analysis, demonstrating the effectiveness of the fusion method combining MHSA and additive attention modules in this invention.

[0132] Step 5: Input the sample to be predicted into the trained collaborative learning multimodal emotion recognition model and output the corresponding predicted emotion state.

[0133] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods.

[0134] Example 2:

[0135] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0136] Example 3:

[0137] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0138] Example 4:

[0139] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0140] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A multimodal emotion recognition method based on attention mechanism under collaborative learning, characterized in that, Includes the following steps: Step 1: Obtain samples and corresponding emotional states. Generate training, validation, and test sets based on the samples. Samples include facial images and corresponding text inputs. Text inputs include Token IDs, attention masks, and paragraph labeling results. Step 2: Construct a collaborative learning multimodal emotion recognition model. The collaborative learning multimodal emotion recognition model includes a feature extractor, a feature fusion module, and a classifier. The sample extracts features through the feature extractor, then fuses them through the feature fusion module, and finally classifies them using the classifier. The feature fusion module includes multi-head self-attention and additive attention modules. Step 3: Define the loss function for the collaborative learning multimodal emotion recognition model; Step 4: Train the collaborative learning multimodal emotion recognition model based on the training set; Step 5: Input the sample to be predicted into the trained collaborative learning multimodal emotion recognition model, and output the corresponding predicted emotion state. In step 2, the feature extractors include the BERT model and the VIT model. The BERT model extracts text input... Text features are obtained by feature extraction. VIT model for facial images Image features are obtained by extracting features. Use fully connected layers to align text features and image features Feature dimensions, to obtain features and characteristics , The feature fusion module in step 2 includes multi-head self-attention and additive attention modules, which concatenate features. and characteristics Preliminary fusion feature Fc is obtained: ； Obtain the weight matrix of the self-attention head, including the query matrix. Key matrix Value matrix , ； Where · denotes matrix multiplication. , , All of these are learnable parameter matrices. Calculate the weighted output of the self-attention head : ； Among them, feature dimension It is a query matrix Bond matrix The number of columns, It is a matrix The transpose of , Softmax() is the Softmax function, The weighted outputs of each self-attention point are concatenated to obtain the fused feature. : ； in, It is the output weight matrix. It is the number of self-focused heads. Get the new query matrix and new key matrix : ； in, , For a learnable parameter matrix, Calculate additive attention weights : ； in, It is a learnable weight vector. It is a scaling factor. This represents scalar multiplication. Calculate normalized attention weights : ； in, Represents additive attention weights along Perform normalization operation. Calculate the global context vector : ； Calculate weighted attention output : ； in, For the projection function, To query vector The representation space projection it is located in. Weighted attention output As input, the data is mapped to the category space through a fully connected layer, and the predicted classification probability is output using the Softmax function.

2. The multimodal emotion recognition method based on attention mechanism under collaborative learning according to claim 1, characterized in that, The emotional states in step 1 include positive leadership, positive dependence, positive relaxation, positive irrelevance, negative hostility, negative anxiety, negative contempt, negative irrelevance, and neutrality.

3. The multimodal emotion recognition method based on attention mechanism under collaborative learning according to claim 1, characterized in that, The sample in step 1 is obtained based on the following steps; Text data of participants' speeches and personal facial video data during the speaking time were collected as raw individual data. Each raw individual data point corresponds to an emotional state. Personal images are extracted from personal facial video data at set time intervals, and personal images are denoising through a Gaussian filter. Then, face detection and cropping are performed to obtain facial images. The obtained facial images are uniformly set in pixel size and data augmentation is performed. Text data is segmented to obtain text input.

4. The multimodal emotion recognition method based on attention mechanism under collaborative learning according to claim 1, characterized in that, The loss function of the collaborative learning multimodal emotion recognition model is based on the following formula: ； in, Indicates the first Is the true label of the sample the first one? Emotional states, Indicates the first The sample was classified as the first... Predicted classification probability of emotion states Represents batch size, This represents the total number of emotional states.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the identification method according to any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the identification method according to any one of claims 1 to 4.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the identification method according to any one of claims 1 to 4.

Images