A multi-modal sentiment recognition method, system, device and medium based on self-attention mechanism fusion

Abstract

A multi-modal sentiment recognition method, system, device and medium based on self-attention mechanism fusion, the method comprising: data preprocessing; feature extraction; feature fusion; model training; task migration; system, device and medium: for realizing a multi-modal sentiment recognition method based on self-attention mechanism fusion; the pre-training model is fine-tuned by the method of contrast learning, so as to obtain the correlation information between modes, and the self-attention mechanism is used for feature fusion, and three different transfer functions are designed to adapt to different downstream tasks, so as to improve the accuracy of the model, improve the transferability and robustness of the algorithm, and make the application range of the model wider.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition technology, and in particular to a multimodal emotion recognition method, system, device and medium based on self-attention mechanism fusion. Background Technology

[0002] As an integral part of human verbal and nonverbal communication, emotion is a key driver of human behavior and thought. Emotion recognition has significant implications for customer service, healthcare, education, and intelligent automotive driving systems. Automatic emotion recognition can not only assist physicians in diagnosis and treatment but also enhance the interactivity of online education. Furthermore, the significant advantage of deep learning in extracting high-dimensional information makes deep learning-based emotion recognition technology a current hot topic with enormous development potential. However, the factors in emotion recognition are diverse. Considering only visual or linguistic information to identify emotional states will result in the loss of a large amount of relevant information, leading to low accuracy. Therefore, it is necessary to consider multiple factors. However, due to the significant differences in data across different categories, it is necessary to fuse features from multiple categories, which greatly increases the difficulty of algorithm design. The massive computational cost of deep network models also imposes many limitations on the practical application scenarios of algorithms, and the diversity of emotion tasks places higher demands on the transferability of algorithms.

[0003] Patent application number [CN202211092432.0] entitled "Brain-like Multimodal Emotion Recognition Network, Recognition Method and Emotion Robot" discloses a brain-like multimodal emotion recognition network and recognition method. The specific steps of the method are as follows: First, extract video facial expression features and video-related audio features, splice the original features, then input them into a bidirectional long short-term memory network, splice all features to obtain a unified feature vector, and perform emotion classification and recognition based on this vector.

[0004] Patent application number [CN202211392299.0] entitled "Multimodal Emotion Recognition Method for Speech and Facial Expression Based on Weighted Feature Fusion" discloses a multimodal emotion recognition method for speech and facial expression based on weighted feature fusion. The emotion recognition steps of this method mainly include: firstly, extracting information from speech keyframes and facial expression keyframes and aligning them, then weighting the extracted features and assigning weights; secondly, using a trained residual network to extract high-order speech emotion features from speech MFCC features, and using a trained 3D-CNN neural network to extract high-order facial expression emotion features from facial expression keyframes; finally, inputting the weighted high-order speech emotion features and the weighted high-order facial expression emotion features into a multilayer perceptron (MLP) for weighted feature fusion and emotion classification.

[0005] However, existing technologies have the following problems:

[0006] (1) In the prior art, the multimodal network structure is too large to be applied on ordinary machine platforms;

[0007] (2) Most existing technologies adopt data-level and decision-level fusion schemes. The former cannot extract effective correlation information, while the latter loses too much correlation information and cannot achieve effective feature fusion.

[0008] (3) Most existing multimodal fusion solutions choose to assign weights to each modality, which cannot adaptively assign weights for different scenarios;

[0009] (4) Most existing multimodal emotion recognition solutions are based on a single task based on a dataset, which cannot be adapted to complex downstream tasks. Summary of the Invention

[0010] To overcome the shortcomings of the prior art, the present invention aims to provide a multimodal emotion recognition method, system, device, and medium based on self-attention mechanism fusion. It fine-tunes the pre-trained model through contrastive learning to obtain correlation information between modalities, and performs feature fusion using self-attention mechanism. At the same time, three different transfer functions are designed to adapt to different downstream tasks. While improving the accuracy of the model, it also improves the transferability and robustness of the algorithm, making the model more widely applicable.

[0011] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0012] A multimodal emotion recognition method based on self-attention mechanism fusion includes the following steps:

[0013] Step 1, Data Preprocessing: Select different time series according to different transfer functions, sample data from multiple modalities, and align them according to the sequence. The multiple modalities of data include text information in text data, speech information in speech data, pose information in video data, and facial information in video data.

[0014] Step 2, Feature Extraction: Input the data of multiple modalities after preprocessing in Step 1 into the corresponding feature extraction network to extract features and obtain the corresponding feature vectors;

[0015] Step 3, Feature Fusion: The self-attention mechanism model is used to assign different weights to the feature vectors of multiple modalities extracted in Step 2, and then feature fusion is performed to obtain normalized sentiment feature values.

[0016] Step 4, Training the model: Use the contrastive learning mechanism to train the multimodal emotion recognition network model based on self-attention mechanism fusion. Repeat steps 1-4 to optimize the extracted feature vectors and obtain the optimal multimodal emotion recognition network model based on self-attention mechanism fusion.

[0017] Step 5, Task Transfer: The transfer module is used to transfer different downstream tasks. Different transfer functions are set according to three different emotional states: absolute emotion, relative emotion, and emotional tendency. This is used to process the normalized emotional feature values ​​obtained after training the multimodal emotion recognition network model based on the self-attention mechanism in Step 4, and to obtain the emotional values ​​of different tasks.

[0018] The specific process of feature extraction in step 2 is as follows:

[0019] Step 2.1: Extract feature vectors of text sentiment information from text data, specifically including: cropping text segments by setting a time series; embedding word vectors of text segments to obtain a sequence of corresponding position information; inputting the sequence of corresponding position information into the BERT network in the feature extraction network to obtain feature vectors of text sentiment information;

[0020] Step 2.2: Extract feature vectors of speech emotion information from speech data, specifically including: extracting speech segments through a set time series; performing Fourier transform on the speech data of each speech segment to obtain a one-to-one corresponding signal spectrum; obtaining the Mel-wave cepstral spectrum by passing the obtained signal spectrum through Spec_transition; and inputting the obtained Mel-wave cepstral spectrum into the GRU network in the feature extraction network to obtain the feature vector of speech emotion information.

[0021] Step 2.3: Extract feature vectors of pose and emotion information from video data, specifically including: extracting image frames from video data according to a set time sequence; performing skeleton detection on each acquired image using the Alphapose network to obtain human pose skeleton information for each image; vectorizing the human pose skeleton information; and using convolutional layers to extract features from the human pose skeleton information to obtain feature vectors of pose and emotion information.

[0022] Step 2.4: Extract feature vectors of facial emotion information from video data, specifically including: extracting image frames from video data according to a set time sequence; performing face detection on each acquired image using the YOLO network architecture; obtaining face coordinate information for each image; cropping the image using the obtained face coordinate information to obtain face image blocks, and standardizing their size and resolution; inputting the standardized face image blocks into the OpenFace2 library to obtain feature vectors of facial emotion information.

[0023] The specific process of feature fusion in step 3 is as follows:

[0024] Step 3.1: Concatenate the feature vectors of text sentiment information, speech sentiment information, posture sentiment information, and facial sentiment information obtained in Step 2.

[0025] Step 3.2: Adaptively control the weights of modal information through a self-attention mechanism, and then concatenate them using the Concat function to obtain feature data that integrates text, speech, and video;

[0026] Step 3.3: Input the feature data of the fused text, voice and video into the fully connected module, and output the normalized sentiment feature value.

[0027] The specific process of training the model in step 4 is as follows:

[0028] Step 4.1: Divide the data of the multiple modalities aligned in Step 1 into training and test sets;

[0029] Step 4.2: Divide the training set from Step 4.1 into different samples according to the aligned text time series. Modal information within a sample can be mutually established as positive samples, and modal information between different samples can be used as negative samples, thereby obtaining the positive and negative samples required for model training.

[0030] Step 4.3: Extract and fuse features from the positive and negative samples obtained in Step 4.2 in sequence, and use the comparison relationship between the positive and negative samples as the input parameter of the loss function to adjust the ability of each network to extract relevant information. Repeat Steps 1-4 to train and optimize the multimodal emotion recognition network model based on self-attention mechanism fusion, and obtain the optimal multimodal emotion recognition network model based on self-attention mechanism fusion.

[0031] The specific process of task migration in step 5 is as follows:

[0032] Step 5.1: Classify emotions into three states based on continuous and discrete emotion categories, including absolute emotion, relative emotion, and emotion tendency;

[0033] Step 5.2: Set different transfer functions according to the three different emotional states to process the normalized emotional feature values ​​obtained after training in Step 4;

[0034] Step 5.3: Input the results of Step 5.2 into ReLU and fully connected layers to obtain sentiment values ​​for different tasks.

[0035] In the feature fusion process of step 3, the specific method for adaptively controlling the modal information weights through the self-attention mechanism is as follows:

[0036] By utilizing a self-attention mechanism to adjust matrix parameters to obtain the correlation between the annotation results and the input vector, the matrix is ​​guided to perform adaptive weight allocation. The resulting weighted sum is the ranking weighted sum of sentiment relevance, thereby achieving weight allocation for multiple modalities. The specific formula is as follows:

[0037] a' 1,i =exp(a 1,i ) / ∑ j exp(a 1,j )

[0038] In the formula, i represents the i-th mode, j represents the number of modes, and a 1,i a represents the weight of the i-th mode. 1,j This represents the weight of the j-th mode.

[0039] The transfer functions in step 5 include absolute emotion transfer functions, relative emotion transfer functions, and emotion tendency transfer functions. These three types of transfer functions can be adapted to different downstream tasks.

[0040] The absolute emotion transfer function is:

[0041]

[0042] In the formula, g a Let w represent the base value of the median, w represent the number of samples in the sequence, i represent the i-th sample, and v represent the base value of the median. i This represents the value of the sample.

[0043] The relative sentiment transfer function is:

[0044]

[0045] In the formula, c a The median effect change is represented by w, where w represents the number of samples in the sequence, i represents the i-th sample, and v i This represents the value of the sample, v. i-1 This represents the value of the previous sample;

[0046] The sentiment transfer function is:

[0047]

[0048] In the formula, t a The median influences the trend value, w represents the number of samples in the sequence, i represents the i-th sample, and v i This represents the value of the sample, v. i-1 This represents the value of the previous sample.

[0049] A multimodal emotion recognition system based on self-attention mechanism fusion includes:

[0050] Data preprocessing module: Selects different time series based on different transfer functions, samples data from multiple modalities, and aligns the data according to the sequence;

[0051] Feature extraction module: Uses a pre-trained model to extract the sentiment information contained in the pre-processed data from multiple modalities and obtain the corresponding feature vectors;

[0052] Feature fusion module: The self-attention mechanism model is used to fuse the feature vectors of multiple modalities to obtain normalized sentiment feature values;

[0053] Model training module: The preprocessed data from multiple modalities are used to generate corresponding positive and negative samples through a contrastive learning mechanism. The extracted feature vectors are optimized and then input into the multimodal emotion recognition network model based on self-attention mechanism fusion. Feature extraction and feature fusion are performed sequentially to train the model and obtain the optimal multimodal emotion recognition network model based on self-attention mechanism fusion.

[0054] Task transfer module: Different transfer functions are set according to three different emotional states to process the normalized emotional feature values ​​obtained after training the multimodal emotion recognition network model based on self-attention mechanism, and output two-dimensional emotional values.

[0055] A multimodal emotion recognition device based on self-attention mechanism fusion includes:

[0056] Memory: Used to store the computer program that implements the multimodal emotion recognition method based on self-attention mechanism fusion;

[0057] Processor: Used to implement the self-attention mechanism-based multimodal emotion recognition method when executing the computer program.

[0058] A computer-readable storage medium comprising:

[0059] The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of a multimodal emotion recognition method based on self-attention mechanism fusion.

[0060] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0061] 1. In step 1 of this invention, relevant information from text data, voice data and video data are collected respectively, and the text information, voice information, voice signal, posture information and facial information in the relevant data are fully considered to ensure the diversification of data sources.

[0062] 2. Step 3 of this invention fuses the feature vectors of multiple modalities, thereby improving the accuracy of the training model.

[0063] 3. In step 3 of this invention, the weights of multimodal information are adaptively controlled through a self-attention mechanism, thereby achieving adaptive fusion of information from multiple modalities and improving the accuracy and generalization of the model.

[0064] 4. In step 4 of this invention, the pre-trained model is fine-tuned by using comparative learning, which greatly reduces the number of model parameters and the training cost of the model.

[0065] 5. In step 5 of this invention, the transfer module is used to transfer different downstream tasks, which improves the robustness of the model.

[0066] In summary, compared with existing technologies, this invention fully considers text information, speech information, speech signals, posture information, and facial information, ensuring the diversity of data sources; it also employs feature-level fusion of the pre-trained model, which greatly reduces the number of model parameters, improves the model's processing speed, and expands the model's application scenarios; at the same time, it uses a self-attention mechanism for feature fusion, combined with different transfer functions under the contrastive learning framework, which improves the model's accuracy while enhancing the algorithm's transferability and robustness, making the model's application scope wider. Attached Figure Description

[0067] Figure 1 This is a flowchart of the method of the present invention.

[0068] Figure 2 This is a diagram of the overall architecture of the present invention.

[0069] Figure 3 This is a diagram of the Transformer encoder architecture in the BERT network of the feature extraction network of this invention.

[0070] Figure 4 This is a diagram of the GRU network architecture in the feature extraction network of this invention.

[0071] Figure 5 This is a diagram of the Alphapose network architecture in the feature extraction network of this invention.

[0072] Figure 6 This is a diagram of the YOLO network architecture in the feature extraction network of this invention.

[0073] Figure 7 This is a diagram of the self-attention mechanism model architecture of the present invention.

[0074] Figure 8 This is a schematic diagram of the comparative learning framework of the present invention.

[0075] Figure 9 This is a horizontal comparison of the results of different fusion methods. Detailed Implementation

[0076] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0077] See Figure 1 A multimodal emotion recognition method based on self-attention mechanism fusion includes the following steps:

[0078] Step 1, Data Preprocessing: Select different time series according to different transfer functions, sample data from multiple modalities, and align them according to the sequence. The multiple modalities of data include text information in text data, speech information in speech data, pose information in video data, and facial information in video data.

[0079] Step 2, Feature Extraction: Input the data of multiple modalities after preprocessing in Step 1 into the corresponding feature extraction network to extract features and obtain the corresponding feature vectors;

[0080] Step 3, Feature Fusion: The self-attention mechanism model is used to assign different weights to the feature vectors of multiple modalities extracted in Step 2, and then feature fusion is performed to obtain normalized sentiment feature values.

[0081] Step 4, Training the model: Use the contrastive learning mechanism to train the multimodal emotion recognition network model based on self-attention mechanism fusion. Repeat steps 1-4 to optimize the extracted feature vectors and obtain the optimal multimodal emotion recognition network model based on self-attention mechanism fusion.

[0082] Step 5, Task Transfer: Transfer functions are used to transfer different downstream tasks. Different transfer functions are set according to three different emotional states: absolute emotion, relative emotion, and emotional tendency. This is used to process the normalized emotional feature values ​​obtained after training the multimodal emotion recognition network model based on the self-attention mechanism in Step 4, and to obtain the emotional values ​​of different tasks.

[0083] See Figure 2 As an overall algorithm framework, the BERT network model is a text feature extraction network, the GRU network model is a speech feature extraction network, the Alphapose network model is a pose feature extraction network, and the YOLO+Openface2 network model is a facial expression feature extraction network. Feature vectors are extracted through attention mechanism, training is completed using contrastive learning framework, and transfer functions are used to complete the transfer to downstream tasks.

[0084] Specifically, the CMU_MOSEI dataset was selected for data preprocessing. This dataset has two labels: sentiment and emotion. Sentiment is defined as seven stages from negative to positive, ranging from -3 to 3. Emotion includes seven categories: anger, happiness, sadness, surprise, fear, and disgust. For several modalities such as text, speech, and video, data was captured and aligned at fixed frequencies to form computational sequences for subsequent training. During data preprocessing, sentiment was divided into seven categories based on the labels, and emotion was a tendency value of [-3, 3]. Different time series were selected for sampling text, speech, and video data according to different transfer functions, and then aligned according to the sequences.

[0085] The aligned data was divided into training and testing sets in a 7:3 ratio. After training the multimodal network for 500 epochs, the model gradually regressed to a stable optimal solution, effectively extracting the four corresponding sentiment-related features. Then, a self-attention mechanism was used to adaptively assign different weights to different modalities, achieving effective feature-level fusion.

[0086] See Figure 3 , Figure 4 , Figure 5 and Figure 6 The specific process of feature extraction in step 2 is as follows:

[0087] Step 2.1: Extract feature vectors of text sentiment information from text data, specifically including: cropping text segments by setting a time series; embedding word vectors of text segments to obtain a sequence of corresponding position information; inputting the sequence of corresponding position information into the BERT network in the feature extraction network to obtain feature vectors of text sentiment information;

[0088] Step 2.2: Extract feature vectors of speech emotion information from speech data, specifically including: extracting speech segments through a set time series; performing Fourier transform on the speech data of each speech segment to obtain a one-to-one corresponding signal spectrum; obtaining the Mel-wave cepstral spectrum by passing the obtained signal spectrum through Spec_transition; and inputting the obtained Mel-wave cepstral spectrum into the GRU network in the feature extraction network to obtain the feature vector of speech emotion information.

[0089] Step 2.3: Extract feature vectors of pose and emotion information from video data, specifically including: extracting image frames from video data according to a set time sequence; performing skeleton detection on each acquired image using the Alphapose network to obtain human pose skeleton information for each image; vectorizing the human pose skeleton information; and using convolutional layers to extract features from the human pose skeleton information to obtain feature vectors of pose and emotion information.

[0090] Step 2.4: Extract feature vectors of facial emotion information from video data, specifically including: extracting image frames from video data according to a set time sequence; performing face detection on each acquired image using the YOLO network architecture; obtaining face coordinate information for each image; cropping the image using the obtained face coordinate information to obtain face image blocks, and standardizing their size and resolution; inputting the standardized face image blocks into the OpenFace2 library to obtain feature vectors of facial emotion information.

[0091] Step 2.1 utilizes the BERT network to extract feature vectors of text sentiment information. Specifically, the BERT network uses multiple Transformer encoders stacked together. The Transformer encoder architecture is as follows: Figure 3 As shown, when performing feature extraction, instruction embedding is first performed on all words in the input, including normal and special words. Then, paragraph embedding is used to distinguish between two sentences. Finally, position embedding is performed. Compared with the sine and cosine functions of Transformer, this invention uses random initialization to update model parameters and uses multi-head attention mechanism and residual network to improve the model's extraction capability.

[0092] In step 2.2, the GRU network is used to extract feature vectors of speech emotion information, such as... Figure 4 As shown. Specifically, compared to the traditional LSTM model, the GRU network structure has only two gates: an update gate and a reset gate. The update gate z... t Information is filtered using the following formula, with weight matrix W. z To control the degree to which new and old information is retained at time t:

[0093] z t =σ(W z ·[h t-1 ,x t ])

[0094] In the formula, z t Indicates an update gate, W z Let represent the weight matrix, σ represent the sigmoid loss function, t represent the current time step, t-1 represent the previous time step, and h represent the weight matrix. t-1 Let x represent the hidden state at the previous time step. t This represents the input sequence at the current time.

[0095] r t The reset gate is also used to filter information, and the weight matrix W is used according to the following formula. r To control the degree to which input information is retained at each location at time t-1:

[0096] r t=σ(W r ·[h t-1 ,x t ])

[0097] In the formula, r t Indicates resetting the door, W r Let represent the weight matrix, σ represent the sigmoid loss function, t represent the current time step, t-1 represent the previous time step, and h represent the weight matrix. t-1 Let x represent the hidden state at the previous time step. t This represents the input sequence at the current time.

[0098] Through the control of these two gates, GRU can dynamically balance the addition of new and old information. Compared with the traditional LSTM model, GRU network has fewer parameters, requiring only 6 weight matrices. Furthermore, it can perform memory selection according to the following formula to filter out important hidden information:

[0099]

[0100]

[0101] In the formula, Let r represent the candidate hidden state, W represent the trainable weight matrix, and r represent the hidden state. t Indicates resetting the door, h t-1 Let x represent the hidden state at the previous time step. t h represents the input of the input sequence at the current time. t z represents the hidden state at the current moment. t This indicates that the door is being reset.

[0102] In step 2.3, the Alphapose network is used to extract feature vectors of pose sentiment information. The Alphapose network architecture is as follows: Figure 5 As shown, specifically, the STN spatial transformation network is used to receive human region proposals. After SPPE, the obtained pose is mapped onto the original human proposal image. Next, the spatial inverse transform network SDTN is used to remap the estimated human pose back to the original image coordinates. At the same time, the SPPE pose estimation network is trained, and data augmentation is performed through the PGPG pose-guided region box generator. Finally, parameterized non-maximum suppression (P-NMS) is used to solve the pose redundancy detection problem and obtain the feature vector of the corresponding pose sentiment information. Through this operation, the human region can be accurately located and the corresponding pose features can be extracted.

[0103] Step 2.4 utilizes the YOLO network to extract feature vectors of facial emotion information. Specifically, firstly, facial expression information is extracted. After obtaining the pre-processed and aligned facial image, the image is input into the YOLO network to obtain facial keypoint information. Then, the Openface2 module is called to perform emotion information analysis, suppressing irrelevant information and maximizing the extraction of emotion-related information while preventing the pooling of emotion-related information. The YOLO network architecture is as follows: Figure 6 As shown, the input is obtained through Mosaic data augmentation, then adaptively scaled and padded with grayscale (114, 114, 114), turning the original 640*640*3 image into the Focus module. In this module, a value is obtained for every single pixel, resulting in four images. Compared to the original RGB three channels, the data now becomes a 320*320*12 feature map. Next, a convolutional layer is used to obtain a 320*320*32 feature map, thereby improving computational power while preserving information. In the CSP module, the data is divided into two branches: one branch undergoes convolution after residual processing, while the other branch is directly convolved. The two branches are then concatenated after passing through a normal distribution and this process is repeated multiple times to finally extract the feature vector of facial emotion information.

[0104] See Figure 7 The specific process of feature fusion in step 3 is as follows:

[0105] Step 3.1: Concatenate the feature vectors of text sentiment information, speech sentiment information, posture sentiment information, and facial sentiment information obtained in Step 2.

[0106] Step 3.2: Adaptively control the weights of modal information through a self-attention mechanism, and then concatenate them using the Concat function to obtain feature data that integrates text, speech, and video;

[0107] Step 3.3: Input the feature data of the fused text, voice and video into the fully connected module, and output the normalized sentiment feature value.

[0108] The self-attention mechanism model mainly consists of two core parts, aiming to calculate the correlation between input vectors using trained matrices. Specifically: First, for the input vectors, different matrices are used to obtain the query matrix and key matrix, and the inner product of q and k is calculated, which measures the degree of correlation between the two vectors. Based on the correlation information, the attention score a' is obtained using the following formula:

[0109] a' 1,i =exp(a 1,i ) / ∑ j exp(a 1,j)

[0110] In the formula, i represents the i-th mode, j represents the number of modes, and a 1,i a represents the weight of the i-th mode. 1,j This represents the weight of the j-th mode;

[0111] Then, after passing through a softmax layer, a W is used. v The matrix is ​​used to score attention and extract information from it, calculating the input vector and W. v The weighted sum of the vectors obtained after matrix multiplication yields adaptive weight values, thereby realizing the weight allocation of multiple modalities.

[0112] See Figure 8 The specific process of training the model in step 4 is as follows:

[0113] Step 4.1: Divide the data of the multiple modalities aligned in Step 1 into training and test sets;

[0114] Step 4.2: Divide the training set from Step 4.1 into different samples according to the aligned text time series. Modal information within a sample can be mutually established as positive samples, and modal information between different samples can be used as negative samples, thereby obtaining the positive and negative samples required for model training.

[0115] Step 4.3: Extract and fuse features from the positive and negative samples obtained in Step 4.2 in sequence, and use the comparison relationship between the positive and negative samples as the input parameter of the InfoNCE loss function to adjust the ability of each network to extract relevant information. Repeat Steps 1-4 to train and optimize the multimodal emotion recognition network model based on self-attention mechanism fusion, and obtain the optimal multimodal emotion recognition network model based on self-attention mechanism fusion.

[0116] Specifically, the training process based on contrastive learning is first divided into two parts: task replacement and objective function, which includes the following four stages:

[0117] (1) Task replacement phase: For a sample x in the same sequence, two different task methods are used to generate and Two samples were used, with random cropping, random color distortion, and Gaussian blur applied to the image data portion, resulting in... and To become a positive sample pair;

[0118] (2) Feature encoder stage: f(·), different networks can be selected for different data. The pre-trained network corresponding to the data of the above multiple modalities is the feature encoder.

[0119] (3) Multilayer perceptron stage: After the above feature extraction, the data enters the multilayer perceptron and outputs the corresponding feature z. i and z j Regression is performed based on the InfoNCE loss function below;

[0120] (4) Objective function stage: InfoNCE loss function on z i and z j The loss is defined as follows:

[0121]

[0122] In the formula, N represents the number of samples in a batch. That is, for N samples in a batch, N pairs of positive samples are obtained through data augmentation, resulting in 2N samples. The negative samples are the remaining 2(N-1) samples, meaning that the negative samples are all generated based on the data in this batch; i Indicates sample x i Features, z j Indicates sample x j Features, z k sim(z) represents the features extracted from the generated 2N samples. i ,z k The formula for calculating cosine similarity is sim(u,v) = u. T v / ||u||·||v||),1[k≠i], input 0 or 1, when k and i are not equal, the output result is 1, otherwise it is 0, τ is the temperature coefficient, used to control the training amplitude.

[0123] Then, loss regression is performed based on the InfoNCE loss function. The numerator calculates the distance between positive sample pairs, while the distance between negative samples is added in the denominator. The closer the positive sample pairs are, the farther the negative sample pairs are, which improves the correlation between the extracted features between modes. The feature extraction function is then fine-tuned based on this loss to complete the feature extraction training of the overall multimodal network.

[0124] The specific process of task migration in step 5 is as follows:

[0125] Step 5.1: Classify emotions into three states based on continuous and discrete emotion categories, including absolute emotion, relative emotion, and emotion tendency;

[0126] Step 5.2: Set different transfer functions according to the three different emotional states to process the normalized emotional feature values ​​obtained after training in Step 4;

[0127] Step 5.3: Input the results of Step 5.2 into ReLU and fully connected layers to obtain sentiment values ​​for different tasks.

[0128] Based on the above measures, this invention explores three different positive / negative sample selection transfer strategies. It should be noted that for all proposed contrastive labeling strategies, the same loss function is used to train the multimodal model, namely the supervised contrastive loss function InfoNCE mentioned above. The three transfer strategies are as follows:

[0129] (1) Absolute Sentiment: Contrastive sentiment is constructed by matching windows with similar sentiment states as positive pairs and windows with different sentiment states as negative pairs. To define the similarity of sentiment states, this invention uses sentiment-supervised contrastive learning for sentiment modeling. Specifically, sentiment states are binaryed into "high" and "low," and windows with the same or different states are considered similar or dissimilar, respectively. The criterion for binarying is based on the median basis truth g of the entire influence annotation trace set. a And a threshold e, when the influence annotation value is greater than g a When a threshold e is added, a time window is marked as "high" when the affected annotation value is less than the median base truth value g. a Subtracting the threshold e results in a value labeled "low". The threshold e is a window used to eliminate ambiguous sentiment values ​​that are close to the median, which can reduce model stability and thus affect the effectiveness of learned features. The resulting preprocessed dataset does not include ambiguous sentiment values ​​and forms the basis for all three labeling strategies. a As shown in the formula below, v i That is, the value of the sample:

[0130]

[0131] In the formula, g a Let w represent the base value of the median, w represent the number of samples in the sequence, i represent the i-th sample, and v represent the base value of the median. i This represents the value of the sample.

[0132] (2) Relative Sentiment: By selecting the median for binaryization, a balanced dataset can be obtained. Furthermore, this invention employs a binarization procedure based on relative measurements of sentiment changes, where the influence annotation value is greater than the median influence change value c. a When the threshold e is added, a time window is selected to be marked as "change" when the influence annotation value is less than the median influence change value c. a When the threshold e is subtracted, it is marked as "unchanged":

[0133]

[0134] In the formula, c a The median effect change is represented by w, where w represents the number of samples in the sequence, i represents the i-th sample, and vi This represents the value of the sample, v. i-1 This represents the value of the previous sample;

[0135] (3) Sentiment Tendency: Similarly, sentiment tendency can also be used for relative measurement. When the influence annotation value is greater than the median influence trend value t... a In this process, a time window is assigned to the "upward trend" category if necessary, and to the "downward trend" category otherwise. Finally, labels are used to define positive and negative samples, and they are matched according to their respective categories. These contrastive labeling strategies can help build a more accurate sentiment model. This invention... The median influence value of the entire trend trajectory is set, and labels are used to define positive and negative samples based on their category: a class match and a class mismatch define positive and negative sample pairs respectively.

[0136]

[0137] In the formula, t a The median influences the trend value, w represents the number of samples in the sequence, i represents the i-th sample, and v i This represents the value of the sample, v. i-1 This represents the value of the previous sample.

[0138] In summary, the main difference between the first strategy and the other two contrast marking strategies is that the former is direct because the values ​​of "high" and "low" are derived from the actual size of the annotation traces; while the other two strategies are indirect because "change" and "trend" are higher-order traces that express the average absolute rate of change and the rate of curvature.

[0139] See Figure 9 Using the Ubuntu 18.04 operating system and VSCODE software, experiments were conducted on different fusion methods. The specific experimental conditions are shown in Table 1.

[0140] Table 1 Experimental conditions

[0141]

[0142]

[0143] First, a horizontal comparison experiment of the fusion methods was conducted. In Table 1, batchsize is the number of samples selected in one training session, dropout is the regularization method, dropout in is the proportion of random drops in the input vector, and dropout out is the proportion of drops during forward propagation. Figure 9It can be seen that decision-level fusion is the least efficient. Although data-level fusion improves efficiency, its final convergence result is still poor. The feature-level fusion scheme using self-attention mechanism is the best for the detection accuracy of the model in this invention.

[0144] Secondly, longitudinal comparative experiments were conducted on multiple models, and the results are shown in Table 2. In the table, TFN, Multi-Net, ICCN, and LMF represent different fusion networks, ACC-2 represents the model's performance on different sentiment categories, F1-2 represents the balance between accuracy and recall for multi-label applications, and CC represents the correlation coefficient. It can be seen that the model of this invention has far fewer parameters than large fusion networks such as TFN. Furthermore, thanks to the updated architecture, the accuracy of sentiment recognition has also been improved to a certain extent, with a more significant improvement in accuracy compared to Multi-Net and ICCN. The improvement in both accuracy and efficiency is particularly noticeable with smaller networks such as LMF.

[0145] Table 2 Experimental Results

[0146]

[0147]

[0148] In summary, this invention discloses a multimodal emotion recognition method based on self-attention mechanism fusion. It extracts emotional features from video, text, and speech data, fine-tunes the pre-trained model through contrastive learning to obtain intermodal correlation information, and leverages the advantages of the self-attention mechanism for feature fusion. Three different transfer functions are designed to adapt to different downstream tasks. The proposed multimodal emotion recognition method can maintain high accuracy with a relatively small number of model parameters.

[0149] A multimodal emotion recognition system based on self-attention mechanism fusion includes:

[0150] Data preprocessing module: Selects different time series based on different transfer functions, samples data from multiple modalities, and aligns the data according to the sequence. This module corresponds to step 1.

[0151] Feature extraction module: This module uses a pre-trained model to extract the sentiment information contained in the pre-processed data from multiple modalities and obtains the corresponding feature vectors. This module corresponds to step 2.

[0152] Feature fusion module: The self-attention mechanism model is used to fuse the feature vectors of multiple modalities to obtain normalized sentiment feature values. This module corresponds to step 3.

[0153] Model training module: The preprocessed data from multiple modalities are used to generate corresponding positive and negative samples through a contrastive learning mechanism. The extracted feature vectors are optimized and then input into the multimodal emotion recognition network model based on self-attention mechanism fusion. Feature extraction and feature fusion are performed sequentially to train the model and obtain the optimal multimodal emotion recognition network model based on self-attention mechanism fusion. This module corresponds to step 4.

[0154] Task transfer module: Different transfer functions are set according to three different emotional states to process the normalized emotional feature values ​​obtained after training the multimodal emotion recognition network model based on self-attention mechanism, and output the two-dimensional emotional value. This module corresponds to step 5.

[0155] A multimodal emotion recognition device based on self-attention mechanism fusion includes:

[0156] Memory: Used to store the computer program that implements the multimodal emotion recognition method based on self-attention mechanism fusion;

[0157] Processor: Used to implement the self-attention mechanism-based multimodal emotion recognition method when executing the computer program.

[0158] A computer-readable storage medium comprising:

[0159] The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of a multimodal emotion recognition method based on self-attention mechanism fusion.

Claims

1. A multimodal emotion recognition method based on self-attention mechanism fusion, characterized in that, Includes the following steps: Step 1, Data Preprocessing: Select different time series according to different transfer functions, sample data from multiple modalities, and align them according to the sequence. The multiple modalities of data include text information in text data, speech information in speech data, pose information in video data, and facial information in video data. Step 2, Feature Extraction: Input the data of multiple modalities after preprocessing in Step 1 into the corresponding feature extraction network to extract features and obtain the corresponding feature vectors; Step 3, Feature Fusion: The self-attention mechanism model is used to assign different weights to the feature vectors of multiple modalities extracted in Step 2, and then feature fusion is performed to obtain normalized sentiment feature values. Step 4, Training the model: Use the contrastive learning mechanism to train the multimodal emotion recognition network model based on self-attention mechanism fusion. Repeat steps 1-4 to optimize the extracted feature vectors and obtain the optimal multimodal emotion recognition network model based on self-attention mechanism fusion. Step 5, Task Transfer: The transfer module is used to transfer different downstream tasks. Different transfer functions are set according to three different emotional states: absolute emotion, relative emotion, and emotional tendency. This is used to process the normalized emotional feature values ​​obtained after training the multimodal emotion recognition network model based on self-attention mechanism in Step 4, and to obtain the emotional values ​​of different tasks. The specific process of task migration in step 5 is as follows: Step 5.1: Classify emotions into three states based on continuous and discrete emotion categories, including absolute emotion, relative emotion, and emotion tendency; Step 5.2: Set different transfer functions according to the three different emotional states to process the normalized emotional feature values ​​obtained after training in Step 4; Step 5.3: Input the results of Step 5.2 into ReLU and fully connected layers to obtain sentiment values ​​for different tasks; The transfer functions in step 5 include absolute emotion transfer functions, relative emotion transfer functions, and emotion tendency transfer functions. These three types of transfer functions can be adapted to different downstream tasks. The absolute emotion transfer function is: In the formula, Indicates the base value of the median. Indicates the number of samples in the sequence. Indicates the first One sample, Indicates the first The value of each sample; The relative sentiment transfer function is: In the formula, This indicates the change in median influence. Indicates the number of samples in the sequence. Indicates the first One sample, Indicates the first The value of each sample. Indicates the first The value of each sample; The sentiment transfer function is: In the formula, This indicates the median influence on the trend value. Indicates the number of samples in the sequence. Indicates the first One sample, Indicates the first The value of each sample. Indicates the first The value of each sample.

2. The multimodal emotion recognition method based on self-attention mechanism fusion according to claim 1, characterized in that, The specific process of feature extraction in step 2 is as follows: Step 2.1: Extract feature vectors of text sentiment information from text data, specifically including: cropping text segments by setting a time series; embedding word vectors of text segments to obtain a sequence of corresponding position information; inputting the sequence of corresponding position information into the BERT network in the feature extraction network to obtain feature vectors of text sentiment information; Step 2.2: Extract feature vectors of speech emotion information from speech data, specifically including: extracting speech segments through a set time series; performing Fourier transform on the speech data of each speech segment to obtain a one-to-one corresponding signal spectrum; obtaining the Mel-wave cepstral spectrum by passing the obtained signal spectrum through Spec_transition; and inputting the obtained Mel-wave cepstral spectrum into the GRU network in the feature extraction network to obtain the feature vector of speech emotion information. Step 2.3: Extract feature vectors of pose and emotion information from video data, specifically including: extracting image frames from video data according to a set time sequence; performing skeleton detection on each acquired image using the Alphapose network to obtain human pose skeleton information for each image; vectorizing the human pose skeleton information; and using convolutional layers to extract features from the human pose skeleton information to obtain feature vectors of pose and emotion information. Step 2.4: Extract feature vectors of facial emotion information from video data, specifically including: extracting image frames from video data according to a set time sequence; performing face detection on each acquired image using the YOLO network architecture; obtaining face coordinate information for each image; cropping the image using the obtained face coordinate information to obtain face image blocks, and standardizing their size and resolution; inputting the standardized face image blocks into the OpenFace2 library to obtain feature vectors of facial emotion information.

3. The multimodal emotion recognition method based on self-attention mechanism fusion according to claim 1, characterized in that, The specific process of feature fusion in step 3 is as follows: Step 3.1: Concatenate the feature vectors of text sentiment information, speech sentiment information, posture sentiment information, and facial sentiment information obtained in Step 2. Step 3.2: Adaptively control the weights of modal information through a self-attention mechanism, and then concatenate them using the Concat function to obtain feature data that integrates text, speech, and video; Step 3.3: Input the feature data of the fused text, voice and video into the fully connected module, and output the normalized sentiment feature value.

4. The multimodal emotion recognition method based on self-attention mechanism fusion according to claim 1, characterized in that, The specific process of training the model in step 4 is as follows: Step 4.1: Divide the data of the multiple modalities aligned in Step 1 into training and test sets; Step 4.2: Divide the training set from Step 4.1 into different samples according to the aligned text time series. The modal information within a sample is mutually established as positive samples, and the modal information between different samples is used as negative samples, thereby obtaining the positive and negative samples required for model training. Step 4.3: Extract and fuse features from the positive and negative samples obtained in Step 4.2 in sequence, and use the comparison relationship between the positive and negative samples as the input parameter of the loss function to adjust the ability of each network to extract relevant information. Repeat Steps 1-4 to train and optimize the multimodal emotion recognition network model based on self-attention mechanism fusion, and obtain the optimal multimodal emotion recognition network model based on self-attention mechanism fusion.

5. A multimodal emotion recognition method based on self-attention mechanism fusion according to claim 3, characterized in that, In the feature fusion process of step 3, the specific method for adaptively controlling the modal information weights through the self-attention mechanism is as follows: By utilizing a self-attention mechanism to adjust matrix parameters to obtain the correlation between the annotation results and the input vector, the matrix is ​​guided to perform adaptive weight allocation. The resulting weighted sum is the ranking weighted sum of sentiment relevance, thereby achieving weight allocation for multiple modalities. The specific formula is as follows: In the formula, Indicates the first One modality, Represents the number of modes. Indicates the first The weight of each modality Indicates the first Weights of each modality.

6. A multimodal emotion recognition system based on self-attention mechanism fusion, used to implement the method of claim 1, characterized in that, include: Data preprocessing module: Selects different time series based on different transfer functions, samples data from multiple modalities, and aligns the data according to the sequence; Feature extraction module: Uses a pre-trained model to extract the sentiment information contained in the pre-processed data from multiple modalities and obtain the corresponding feature vectors; Feature fusion module: The self-attention mechanism model is used to fuse the feature vectors of multiple modalities to obtain normalized sentiment feature values; Model training module: The preprocessed data from multiple modalities are used to generate corresponding positive and negative samples through a contrastive learning mechanism. The extracted feature vectors are optimized and then input into the multimodal emotion recognition network model based on self-attention mechanism fusion. Feature extraction and feature fusion are performed sequentially to train the model and obtain the optimal multimodal emotion recognition network model based on self-attention mechanism fusion. Task transfer module: Different transfer functions are set according to three different emotional states to process the normalized emotional feature values ​​obtained after training the multimodal emotion recognition network model based on self-attention mechanism, and output two-dimensional emotional values.

7. A multimodal emotion recognition device based on self-attention mechanism fusion, characterized in that, include: Memory: for storing a computer program that implements the multimodal emotion recognition method based on self-attention mechanism fusion as described in claims 1-5; Processor: Used to implement the multimodal emotion recognition method based on self-attention mechanism fusion according to claims 1-5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, include: The computer-readable storage medium stores a computer program that, when executed by a processor, enables a multimodal emotion recognition method based on self-attention mechanism fusion as described in claims 1-5.

Images