A multimodal emotion recognition method and system based on contrastive learning and Transformer structure

Abstract

The present invention discloses a multimodal emotion recognition method and system based on contrastive learning and Transformer structure, which includes preprocessing and dividing data samples of a data set into a training set, a test set, and a validation set; using an image feature extraction model and a text feature extraction model to extract features from image data and text data respectively; calculating the image-text contrast loss to align image features with text features; using a multi-layer transformer-encoder to fuse modal features and obtain the result of emotion classification through a softmax function; using a data enhancement method and a random dropout method to construct positive and negative samples of data samples, performing contrastive learning with the original data set features, and optimizing the entire model parameters. The present invention uses a lightweight model to extract features, supplemented by modal alignment and contrastive learning tasks, to reduce computational complexity and improve running speed without affecting model accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of emotion recognition technology, and in particular relates to a multimodal emotion recognition method and system based on contrastive learning and Transformer structure. Background Art

[0002] With the development of computer vision and natural language processing, multimodal sentiment analysis has become a hot research area. To understand and apply data from multiple modalities, many scholars have begun using deep learning-based methods.

[0003] Although the application of Transformer structure models in the field of multimodal sentiment analysis has gradually matured in recent years, many challenges still exist. First, many current methods for multimodal processing still extract features from different modal data and then directly feed them into the Transformer structure for modal fusion, and then perform sentiment classification on the fused feature data. This approach essentially still separates the two modalities in different representation spaces, reducing the generalization ability of some models. Second, the introduction of the Transformer structure has brought better sentiment recognition results, but it has also increased the requirements for computing resources, especially when the amount of data from different modalities is large and more Transformer structures are used. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a multimodal emotion recognition method and system based on contrastive learning and Transformer structure, which uses a lightweight feature extraction model to extract features from different modal data, and then performs modal alignment on the contrastive learning of different modal feature representations to improve its deep understanding ability, and then sends it to the Transformer structure for feature fusion, thereby improving the recognition accuracy of the model and making up for the loss of precision caused by the lightweight feature extraction model.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A multimodal emotion recognition method based on contrastive learning and Transformer structure includes the following steps:

[0007] S1. Select a multimodal dataset with text and images as sample data, preprocess and divide the data samples of the dataset to obtain text data and image data for training and testing.

[0008] S2. Cut the image data into blocks and input them into the image feature extraction model to extract the visual feature representation of the image blocks and obtain the image feature sequence.

[0009] S3. Divide the text data into word vectors, convert them into vector form using word vector mapping, and input them into the text feature extraction model to extract text feature representation and obtain text feature sequence.

[0010] S4. Calculate the image-text contrast loss function for the extracted image feature sequence and text feature sequence to complete the alignment of the feature sequences.

[0011] S5. Concatenate the aligned image feature sequence and text feature sequence to obtain the image-text feature representation, which is then fed into a multi-layer Transformer encoder structure for feature layer fusion to obtain the initial image-text fusion vector representation for subsequent sentiment classification tasks.

[0012] S6. Perform random dropout on the initial image-text fusion vector representation to obtain multiple vector representations. These are then fed into a fully connected layer and passed through a softmax function to obtain the predicted emotion recognition result. The predicted emotion recognition result is then compared with the true emotion category label of the sample to calculate the cross-entropy loss function, thereby optimizing the network parameters.

[0013] S7. Perform data enhancement on the image and text data obtained in step S1. Obtain the image-text fusion vector representation after data enhancement through the operations of steps S2-S5. Compare and learn the image-text fusion vector representation with the initial image-text fusion vector representation obtained in step S5, and calculate the loss function.

[0014] S8. Add the loss functions in steps S4, S6, and S7 to optimize the network parameters in the entire step.

[0015] S9. Input the test sample and execute steps S2-S8 to extract the feature representation corresponding to the sample. The final emotion recognition result is obtained through the fully connected layer and softmax function to complete the emotion recognition task.

[0016] Furthermore, in step S1, the image-text data in the dataset is preprocessed by label: image-text pairs with opposite sentiment labels are deleted from the dataset, and the sentiment labels of the sample data are determined through a voting mechanism. The preprocessed data samples are divided into training, validation, and test sets according to a preset ratio, ensuring that the image and text samples in the divided datasets are equal in size.

[0017] Furthermore, in step S2, the image feature extraction model divides the input image data into fixed-size tiles, which are then converted into vector representations through an embedding layer. The vector representations are then fed into the Transformer structure, where the self-attention mechanism and feedforward neural network are used to generate the image feature representation. The fully connected layer then resizes the sequence to produce the image feature sequence.

[0018] Furthermore, in step S3, the text sequence is divided into independent word sequences. A word vector model is then used to convert the word sequence into a word vector sequence. The resulting word vector sequence is fed into a text feature extraction model, where a multi-head self-attention mechanism and a feedforward neural network are used to obtain feature representations at different levels. This is then integrated to obtain a single text feature sequence, which is then resized using a fully connected layer to obtain a text feature sequence.

[0019] Furthermore, in step S4, the image features and the text features are aligned, specifically including:

[0020] S401: Calculate the similarity between the image feature sequence and the text feature sequence obtained in steps S2 and S3. The specific formula is:

[0021]

[0022] Among them, sim(i,t) represents the similarity score calculation between the image feature sequence and the text feature sequence, CosineSimilarity represents the cosine similarity calculation, i represents the image feature sequence, t represents the text feature sequence, and temp represents the temperature parameter.

[0023] S402: Calculate the loss function between the image feature sequence and the text feature sequence respectively, take the average value, and optimize by maximizing the similarity of positive pairs and minimizing the similarity of negative pairs. The specific formula is:

[0024]

[0025]

[0026] L ITC =(L i2t +L t2i ) / 2;

[0027] Where I represents the set of image feature sequences, T represents the set of text feature sequences, and t + Indicates the positive text that matches the image, i + represents the positive image matching the text, L i2t Represents the loss calculation from image to text, L t2iRepresents the loss calculation from text to image, L ITC Represents the image-text contrast loss, t' represents any text feature sequence in the text feature set, and i' represents any image feature sequence in the image feature set.

[0028] Furthermore, in step S5, the initial image-text fusion vector representation is obtained, including the following contents:

[0029] f I-T =concat(f i ,f t );

[0030] F=transformer(f I-T ).

[0031] Among them, f I-T represents the image-text feature after the image feature and text feature are concatenated, f i represents the image features, f t represents text features, and F represents the initial image-text fusion vector representation.

[0032] Furthermore, step S6 is specifically as follows:

[0033] Random dropout is performed on the initial image-text fusion vector representation to inactivate a portion of the emotional feature content, resulting in an emotional feature sample. The emotional feature is then input into a fully connected layer to obtain a nonlinear transformation, and a softmax calculation is performed to obtain a probability output for emotion classification. Multiple probability outputs are then summed to obtain a single probability output. The class with the highest probability in the probability output is selected as the predicted emotional label for the sample. The multi-class cross entropy loss function is then calculated, using the following formula:

[0034] g=GELU(W g F+b g );

[0035]

[0036]

[0037] Among them, g represents the emotion category feature vector; Represents the probability distribution of emotion categories predicted by the model; N represents the total number of samples; c represents the total number of emotion categories; represents the predicted probability that the mth sample belongs to the nth emotion category; y mn W represents the nth emotion category in the one-hot encoding obtained by converting the mth sample through the classification label. It usually takes a value of 0 or 1, indicating whether the sample m belongs to the nth emotion category; grepresents the weight of the activation function; b g represents the bias of the activation function; W o represents the weight of the output layer; b o represents the bias of the output layer; L c Represents the cross entropy loss function, which is used to calculate the difference between the results predicted by the method and the true label.

[0038] The weights and biases in the above formula are optimized by the Adam optimizer.

[0039] Furthermore, step S7 is specifically as follows:

[0040] S701. Perform data enhancement on the image data. Select any two of the following methods: affine mapping, flipping, cropping, padding, blurring, pixel brightness conversion, contrast adjustment, and noise introduction, perform secondary processing on the image data, and form a positive sample with the original data for comparative learning:

[0041] I a =data_augmentation(I);

[0042] f ia =linear(MobileViT(I a ));

[0043] Among them, I a Represents the image feature sequence set after data enhancement, f ia Represents the image features after feature unification.

[0044] S702: Data augmentation is performed on the text data. The specific method is to perform dropout processing on the selected text data. That is, the same batch of text data is subjected to two feature extraction operations. Through random dropout in the encoder, two different embedding results of the same sentence are obtained. The two results are similar but have their own characteristics. The two vectors from the same text data are each other's positive samples, and the feature vectors of the remaining text data are used as negative samples for comparative learning. The specific formula is:

[0045] f ta = linear(MobileBERT(T));

[0046] Among them, f ta Represents the text features after feature unification.

[0047] S703: Perform feature alignment on the data-enhanced data, then perform feature fusion to obtain the data-enhanced image-text fusion vector representation. Perform contrastive learning and calculate the contrastive learning loss between the fusion features of the original data and the enhanced fusion features. Here, a contrastive learning loss based on angular edge is used to convert the similarity calculation results into angles, and then perform the loss function calculation. The specific formula is:

[0048] L ITC-a =(L' i2t +L' t2i ) / 2;

[0049] F a =transformer(f' I-T );

[0050]

[0051]

[0052] Among them, L ITC-a Represents the image-text feature alignment loss function after data enhancement; L′ i2t Represents the loss calculation from image to text after data enhancement; L′ t2i represents the loss calculation from text to image after data enhancement; f' I-T represents the image-text splicing features after data enhancement; L o-a represents the normalized cross loss between the fused sentiment features without data augmentation and the fused features after data augmentation; Indicates converting the result of cosine similarity calculation into angle value; Indicates the calculation of F and F a The difference in classification results between F represents the emotion recognition result without data enhancement, Indicates the emotion recognition results after data enhancement; Indicates the angle value result of the positive sample; H represents the sample set of the same batch of samples; M represents the set angle bias, F a Represents the image-text fusion vector representation after data enhancement.

[0053] Furthermore, in step S8, the loss functions calculated in steps S4, S6, and S7 are added together to optimize the training parameters. The specific formula is:

[0054] L=L LTC +L c +L LTC-a +L o-a .

[0055] Furthermore, the present invention also proposes a multimodal emotion recognition system based on contrastive learning and Transformer structure, including

[0056] The data extraction module is used to select a multimodal data set with text and images as sample data, preprocess and divide it to obtain text data and image data.

[0057] The image feature sequence acquisition module is used to block the image data and input it into the image feature extraction model to extract the visual feature representation of the image block to obtain the image feature sequence.

[0058] The text feature sequence acquisition module is used to divide text data into word vectors, convert them into vector form using word vector mapping, input them into the text feature extraction model to extract text feature representation, and obtain text feature sequences.

[0059] The feature alignment module is used to calculate the image-text contrast loss for the extracted image feature sequence and text feature sequence to complete the alignment of the feature sequences.

[0060] The initial image-text fusion vector representation acquisition module is used to splice the aligned image feature sequence and text feature sequence to obtain the image-text feature representation, which is then fed into a multi-layer Transformer encoder structure for feature layer fusion to obtain the initial image-text fusion vector representation.

[0061] The cross-entropy loss calculation module performs random dropout on the initial image-text fusion vector representation to obtain multiple vector representations. These representations are then fed into the fully connected layer and used through the softmax function to predict the emotion recognition result. The cross-entropy loss function is then calculated by comparing the predicted emotion recognition result with the true emotion category label of the sample.

[0062] The loss function calculation module is used to perform data enhancement on the image and text data obtained in the data extraction module to obtain the image-text fusion vector representation after data enhancement, compare it with the initial image-text fusion vector representation obtained in the initial image-text fusion vector representation acquisition module, and calculate the loss function.

[0063] The network parameter optimization module is used to add the loss functions in the feature alignment module, cross entropy loss calculation module, and loss function calculation module to optimize the network parameters.

[0064] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0065] The present invention proposes a lightweight model that uses the Bert model and the VIT model. It uses methods such as deep separable convolution and model pruning to reduce the model's calculation and parameter quantity. This reduces the demand for computing resources and improves the running speed while minimizing the loss of model accuracy. The contrastive learning method is used to help the model focus more on the correlation between emotional features in graphic features, optimize the overall accuracy of the model, and compensate for the partial accuracy loss caused by the lightweight model. Compared with the traditional contrastive learning loss calculation, the use of angle calculation enhances the discrimination ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is an overall implementation flow chart of the present invention.

[0067] Figure 2 This is a flow chart of emotion feature extraction and recognition of the present invention.

[0068] Figure 3 It is a flow chart of the implementation of the comparative learning method of the present invention. DETAILED DESCRIPTION

[0069] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0070] To achieve the above objectives, the present invention proposes a multimodal emotion recognition method based on contrastive learning and Transformer structure, such as Figure 1 As shown in Figure 3, the specific steps are divided into emotion feature extraction and recognition and comparative learning.

[0071] The overall implementation process of emotion feature extraction and recognition is as follows: Figure 2 As shown,

[0072] S1. Select a multimodal dataset with text and images as sample data, preprocess and divide it to obtain text data and image data. The specific content is as follows:

[0073] In this example, the selected datasets are MVSA-single and MVSA-mutiple, which are derived from image-text pairs on the Twitter platform. The former contains 4,869 image-text pairs, and the latter contains 19,600 image-text pairs. Labeling of the data samples in these datasets is performed: a voting mechanism is used to select the most frequently occurring sentiment polarity label for multiple labels. The specific partitioning criteria for this dataset are: the dataset is divided into training, validation, and test sets in a specific ratio of 6:2:2, and the data volume of image and text samples in the same data set is kept equal.

[0074] S2. Cut the image data into blocks and input them into the MobileViT model to extract the visual feature representation of the image blocks and obtain the image feature sequence. The specific content is:

[0075] After feature extraction using the MobileViT model, the feature vector is resized through a linear layer to produce image_feat. The MobileViT model is a lightweight version of the VIT model, employing a hybrid architecture of Vit and CNN. The CNN provides the spatial inductive bias lacking in the Transformer architecture, accelerating network convergence and inference. It also incorporates spatial information to mitigate the effects of extra spatial position bias, improving network portability. Furthermore, when calculating self-attention, only the self-attention of tokens at the same position in different patches is calculated, discarding the calculations of other blocks, significantly reducing computational complexity. All image samples are resized, with the default value set to 256, so the input image size is uniformly 256*256. After a 3*3 convolution with a stride of 2, the image is reduced to 128*128. The feature map is then generated through multiple MV2 blocks (or inverted residual structures) with strides of 1 and 2. The feature map is then processed by MobileViTBlock: first, local representation modeling is performed through a 3*3 convolution layer, and then the number of channels is adjusted through a 1*1 convolution layer. The obtained feature map is then globally represented and modeled through splitting, Tranformer block, and folding. The output feature map is again passed through a 1*1 convolution layer to adjust the number of channels. After being spliced ​​with the input feature map along the channel direction, feature fusion output is performed through a 3*3 convolution layer. Finally, the output size is adjusted through a linear layer to obtain a 256-dimensional image feature sequence.

[0076] S3. Divide the text data into word vectors, convert them into vector form using word vector mapping, and input them into the MobileBERT model to extract text feature representation and obtain text feature sequences. The specific content is:

[0077] The text sequence is divided into independent word sequences. Word2Vec is used to convert the word sequence into a word vector sequence. The resulting word vector sequence is fed into MobileBERT to extract the text feature vector. The feature vector is then resized through a linear layer and aligned with the image feature vector of S2 to obtain text_feat. Similar to MobileViT, MobileBERT is also a lightweight model derived from knowledge distillation using BERT. The 256-dimensional word vector sequence output by Word2Vec is reduced in dimension through MobileBERT's unique bottleneck, or linear layer. This reduces the width of each layer, reducing the number of model parameters and computational complexity. Following the multi-head attention mechanism, the feed-forward network is stacked to address the shift in attention parameter ratio caused by the width change, ultimately outputting a 256-dimensional text feature sequence.

[0078] S4. Calculate the image-text contrast loss for the extracted image feature sequence and text feature sequence to complete the alignment of image features and text features. The specific content is as follows:

[0079] S401: Calculate the similarity between the image feature sequence and the text feature sequence obtained in steps S2 and S3. The specific formula is:

[0080]

[0081] Among them, sim(i,t) represents the similarity score calculation between the image feature sequence and the text feature sequence, CosineSimilarity represents the cosine similarity calculation, i represents the image feature sequence, t represents the text feature sequence, and temp represents the temperature parameter.

[0082] S402, respectively calculate the loss function between the image feature sequence and the text feature sequence, take the average value, and optimize by maximizing the similarity of positive pairs and minimizing the similarity of negative pairs. The specific formula is:

[0083]

[0084]

[0085] L ITC =(L i2t +L t2i ) / 2;

[0086] Where I represents the set of image feature sequences, T represents the set of text feature sequences, and t + Indicates the positive text that matches the image, i +represents the positive image matching the text, L i2t Represents the loss calculation from image to text, L t2i Represents the loss calculation from text to image, L ITC Represents the image-text contrast loss, t' represents any text feature sequence in the text feature set, and i' represents any image feature sequence in the image feature set.

[0087] S5. Concatenate the aligned 256-dimensional image feature sequence and the 256-dimensional text feature sequence to obtain the image-text feature representation, which is then fed into a multi-layer Transformer encoder structure for feature layer fusion to obtain the initial image-text fusion vector representation for the subsequent sentiment classification task. The specific content is as follows:

[0088] f I-T =concat(f i ,f t );

[0089] F=transformer(f I-T ).

[0090] Among them, f I-T represents the image-text feature after the image feature and text feature are concatenated, f i represents the image features, f t represents text features, and F represents the initial image-text fusion vector representation.

[0091] S6. Perform random dropout on the initial image-text fusion vector representation to obtain multiple vector representations, which are then fed into the fully connected layer and passed through the softmax function to obtain the predicted emotion recognition result. The network parameters are optimized by calculating the cross entropy loss between the predicted emotion recognition result and the true emotion category label of the sample. The specific content is as follows:

[0092] The initial image-text fusion vector representation is subjected to random dropout processing. In this embodiment, four random dropout processes are used to inactivate part of the emotional feature content, obtaining four versions of emotional feature samples. The four versions of emotional features are then input into the fully connected layer to obtain a nonlinear transformation, and the probability output for emotion classification is obtained through softmax calculation. The multiple probability outputs are then summed to obtain a final probability output. The largest probability category in the probability output is selected as the predicted emotion label of the sample, and then the multi-class cross entropy loss function is calculated. The specific formula is:

[0093] g=GELU(W g F+b g );

[0094]

[0095]

[0096] Among them, g represents the emotion category feature vector; Represents the probability distribution of emotion categories predicted by the model; N represents the total number of samples; c represents the total number of emotion categories; represents the predicted probability that the mth sample belongs to the nth emotion category; y mn W represents the nth emotion category in the one-hot encoding obtained by converting the mth sample through the classification label. It usually takes a value of 0 or 1, indicating whether the sample m belongs to the nth emotion category; g represents the weight of the activation function; b g represents the bias of the activation function; W o represents the weight of the output layer; b o represents the bias of the output layer; L c Represents the cross entropy loss function, which is used to calculate the difference between the results predicted by the method and the true label.

[0097] The weights and biases in the above formula are optimized by the Adam optimizer.

[0098] Compare the learning process as Figure 3 As shown,

[0099] S7: Perform data enhancement on the image and text data obtained in step S1, obtain the image-text fusion vector representation after data enhancement through the operations of steps S2-S5, compare and learn the image-text fusion vector representation with the initial image-text fusion vector representation obtained in step S5, and calculate the loss function. The specific content is:

[0100] S701. Perform data enhancement on the image data and compare it with the original data to form a positive sample for learning:

[0101] I a =data_augmentation(I);

[0102] f ia =linear(MobileViT(I a ));

[0103] Among them, I a Represents the image feature sequence set after data enhancement, f ia Represents the image features after feature unification.

[0104] S702: Perform data augmentation on the text data. Specifically, the selected text data is subjected to dropout processing. Specifically, the same batch of text data is subjected to two feature extraction operations. Random dropout in the encoder is used to obtain two different embedding results for the same sentence. The two results are similar but have their own characteristics. The two 256-dimensional vectors from the same text data are used as positive samples, and the feature vectors of the remaining text data are used as negative samples for comparative learning. The specific formula is:

[0105] f ta =linear(MobileBERT(T)).

[0106] Among them, f ta Represents the text features after feature unification.

[0107] S703: Perform feature alignment on the data-enhanced data, then perform feature fusion to obtain the data-enhanced image-text fusion vector representation. Perform contrastive learning and calculate the contrastive learning loss between the fusion features of the original data and the enhanced fusion features. Here, a contrastive learning loss based on angular edge is used to convert the similarity calculation results into angles, and then perform the loss function calculation. The specific formula is:

[0108] L ITC-a =(L' i2t +L' t2i ) / 2;

[0109] F a =transformer(f' I-T );

[0110]

[0111]

[0112] Among them, L ITC-a Represents the image-text feature alignment loss function after data enhancement; L′ i2t Represents the loss calculation from image to text after data enhancement; L′ t2i represents the loss calculation from text to image after data enhancement; f' I-T represents the image-text splicing features after data enhancement; L o-a represents the normalized cross loss between the fused sentiment features without data augmentation and the fused features after data augmentation; Indicates converting the result of cosine similarity calculation into angle value; Indicates the calculation of F and F a The difference in classification results between Frepresents the emotion recognition result without data enhancement, Indicates the emotion recognition results after data enhancement; Indicates the angle value result of the positive sample; H represents the sample set of the same batch of samples; M represents the set angle bias, F a Represents the image-text fusion vector representation after data enhancement.

[0113] S8. Add the loss functions in steps S4, S6, and S7 to optimize the network parameters in the entire step. The specific content is:

[0114] Construct an emotion recognition model, which includes: a feature extraction module, consisting of steps S2 and S3; a feature alignment module, consisting of step S4; a feature fusion module, consisting of step S5; an emotion recognition module, consisting of step S6; and a contrastive learning module, consisting of step S7. Then, the loss functions calculated in steps S4, S6, and S7 are added together. The specific formula is:

[0115] L=L LTC +L c +L LTC-a +L o-a .

[0116] S9. Input the test sample and execute steps S2-S8 to extract the feature representation corresponding to the sample. The final emotion recognition result is obtained through the fully connected layer and softmax function to complete the emotion recognition task.

[0117] In this embodiment, computation optimization was performed on an Nvidia Tesla T4 GPU using the following parameters: batchsize = 16, 32, 64, learning rate lr = 2e-5, β1 = 0.9, β2 = 0.99, and temp = 0.6.

[0118] The embodiment of the present invention also proposes a multimodal emotion recognition system based on contrastive learning and Transformer structure, including a data extraction module, an image feature sequence acquisition module, a text feature sequence acquisition module, a feature alignment module, an initial image-text fusion vector representation acquisition module, a cross-entropy loss calculation module, a loss function calculation module, a network parameter optimization module and a computer program that can be run on a processor. It should be noted that each module in the above system corresponds to the specific steps of the method provided in the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present invention.

[0119] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multimodal emotion recognition method based on contrastive learning and Transformer structure, characterized by: The following steps are involved: S1. Select a multimodal dataset with text and image as sample data, preprocess and divide it to obtain text data and image data; S2. Slice the image data and input it into the image feature extraction model to extract the visual feature representation of the image block and obtain the image feature sequence; S3. Divide the text data into word vectors, convert them into vector form using word vector mapping, and input them into the text feature extraction model to extract text feature representation and obtain text feature sequence; S4. Calculate the image-text contrast loss function for the extracted image feature sequence and text feature sequence to complete the alignment of the feature sequences; S5. Concatenate the aligned image feature sequence and text feature sequence to obtain image-text feature representation, which is then fed into a multi-layer Transformer encoder structure for feature layer fusion to obtain an initial image-text fusion vector representation. S6. Perform random dropout processing on the initial image-text fusion vector representation to obtain a vector representation, input it into the fully connected layer, and use the softmax function to obtain the predicted emotion recognition result. The cross entropy loss function is calculated by combining the result with the actual emotion category label of the sample; specifically: Perform random dropout processing on the initial image-text fusion vector representation to obtain sentiment feature samples, input the sentiment feature samples into the fully connected layer to obtain a nonlinear transformation, and calculate the probability output for sentiment classification through softmax calculation. The probability output is summed to obtain a probability output; The largest probability category in the probability output is selected as the predicted emotion of the sample, and then the multi-class cross entropy loss function is calculated. The specific formula is: g=GEL(W g F+b g ); Among them, g represents the emotion category feature vector, Represents the predicted probability distribution of emotion categories, N represents the total number of samples, c represents the total number of emotion categories, Indicates the predicted probability that the mth sample belongs to the nth emotion category, y mn W represents the nth emotion category obtained by converting the mth sample into a classification label. g represents the weight of the activation function, b g represents the bias of the activation function, W o represents the weight of the output layer, b o represents the bias of the output layer, L c represents the cross entropy loss function, F represents the initial image-text fusion vector representation; The weights and biases in the above formula are optimized by the adam optimizer; S7, performing data enhancement on the image and text data obtained in step S1, obtaining the image-text fusion vector representation after data enhancement through the operations of steps S2-S5, performing comparative learning on the image-text fusion vector representation obtained in step S5, and calculating the loss function; specifically: S701. Perform data enhancement on the image data and compare it with the original data to form a positive sample for learning: I a =data_augmentation(I); f ia =linear(MobileViT(I a )); Among them, I a Represents the image feature sequence set after data enhancement, f ia Represents the image features after feature unification; S702: Perform dropout processing on the selected text data to obtain different embedding results for the same sentence. Vectors from the same text data are mutually positive samples, and feature vectors of the remaining text data are used as negative samples for comparative learning. The specific formula is: f ta =linear(MobileBERT(T)); Among them, f ta The text features after the features are unified; S703: Perform feature alignment on the data-enhanced data, then perform feature fusion to obtain the image-text fusion vector representation after data enhancement, perform contrastive learning, convert the similarity calculation result into an angle, and calculate the contrastive learning loss between the fusion feature of the original data and the enhanced fusion feature. The specific formula is: L ITC-a =(The i2t +The t2i ) / 2; F a =transformer(f'I-T); Among them, L ITC-a Represents the image-text feature alignment loss function after data enhancement; L′ i2t Represents the loss calculation from image to text after data enhancement; L′ t2i represents the loss calculation from text to image after data enhancement; f' I-T represents the image-text splicing features after data enhancement; L o-a represents the normalized cross loss between the fused sentiment features without data augmentation and the fused features after data augmentation; Indicates converting the result of cosine similarity calculation into angle value; Indicates the calculation of F and F a The difference in classification results between F represents the emotion recognition result without data enhancement, Indicates the emotion recognition results after data enhancement; Indicates the angle value result of the positive sample; H represents the sample set of the same batch of samples; M represents the set angle bias, F a Represents the image-text fusion vector representation after data enhancement; S8, adding the loss functions in steps S4, S6, and S7 to optimize the network parameters; S9. Input the test sample and execute steps S2-S8 to extract the feature representation corresponding to the sample. The final emotion recognition result is obtained through the fully connected layer and softmax function to complete the emotion recognition task.

2. The multimodal emotion recognition method based on contrastive learning and Transformer structure according to claim 1 is characterized in that In step S1, the text data and image data obtained include the following: Delete the image-text pairs with opposite sentiment labels in the dataset, determine the sentiment labels of the sample data through a voting mechanism, complete the preprocessing, and divide the preprocessed data samples into training set, validation set and test set according to the preset ratio. The image samples and text samples in the divided dataset have equal data volume.

3. The multimodal emotion recognition method based on contrastive learning and Transformer structure according to claim 1 is characterized in that In step S2, obtaining the image feature sequence includes the following: The image feature extraction model is used to divide the input image data into fixed-size blocks, which are converted into vector representations through the embedding layer. The vector representations are sent to the Transformer structure, and the feature representation of the image is obtained through the self-attention mechanism and feedforward neural network. The sequence size is adjusted through the fully connected layer to obtain the image feature sequence.

4. The multimodal emotion recognition method based on contrastive learning and Transformer structure according to claim 1, characterized in that In step S3, obtaining a text feature sequence includes the following: The text sequence is divided into independent word sequences, and the word vector model is used to convert the word sequence into a word vector sequence. The obtained word vector sequence is sent to the text feature extraction model, and feature representations at different levels are obtained through the multi-head self-attention mechanism and feedforward neural network. A single text feature sequence is obtained through integration, and the sequence size is adjusted through the fully connected layer to obtain a text feature sequence.

5. The multimodal emotion recognition method based on contrastive learning and Transformer structure according to claim 1, characterized in that: In step S4, aligning the image features with the text features includes the following sub-steps: S401: Calculate the similarity between the image feature sequence and the text feature sequence obtained in steps S2 and S3. The specific formula is: Where sim(i,t) represents the similarity score calculation between the image feature sequence and the text feature sequence, CosineSimilarity represents the cosine similarity calculation, i represents the image feature sequence, t represents the text feature sequence, and temp represents the temperature parameter; S402, respectively calculate the loss function between the image feature sequence and the text feature sequence, take the average value, and optimize by maximizing the similarity of positive pairs and minimizing the similarity of negative pairs. The specific formula is: L ITC =(L i2t +L t2i ) / 2; Where I represents the set of image feature sequences, T represents the set of text feature sequences, and t + Indicates the positive text that matches the image, i + represents the positive image matching the text, L i2t Represents the loss calculation from image to text, L t2i Represents the loss calculation from text to image, L ITC Represents the image-text contrast loss, t' represents any text feature sequence in the text feature set, and i' represents any image feature sequence in the image feature set.

6. The multimodal emotion recognition method based on contrastive learning and Transformer structure according to claim 5, characterized in that: In step S5, the initial image-text fusion vector representation is obtained, which includes the following contents: f I-T =concat(f i ,f t ); F=transformer(f I-T ); Among them, f I-T represents the concatenated image-text features, f i represents the image features, f t Represents text features.

7. The multimodal emotion recognition method based on contrastive learning and Transformer structure according to claim 1, characterized in that: In step S8, the loss functions calculated in steps S4, S6, and S7 are added together. The specific formula is: L=L LTC +L c +L LTC-a +L o-a 。 8. A system for the multimodal emotion recognition method based on contrastive learning and Transformer structure according to any one of claims 1 to 7, characterized in that: include A data extraction module is used to select a multimodal dataset with text and images as sample data, preprocess and divide it to obtain text data and image data; The image feature sequence acquisition module is used to slice the image data and input it into the image feature extraction model to extract the visual feature representation of the image block and obtain the image feature sequence; The text feature sequence acquisition module is used to divide text data into word vectors, convert them into vector form using word vector mapping, and input them into the text feature extraction model to extract text feature representation and obtain text feature sequences; The feature alignment module is used to calculate the image-text contrast loss for the extracted image feature sequence and text feature sequence to complete the alignment of the feature sequences; The initial image-text fusion vector representation acquisition module is used to concatenate the aligned image feature sequence and text feature sequence to obtain the image-text feature representation, which is then fed into a multi-layer Transformer encoder structure for feature layer fusion to obtain the initial image-text fusion vector representation. The cross-entropy loss calculation module is used to perform random dropout processing on the initial image-text fusion vector representation to obtain a vector representation, which is input into the fully connected layer and the softmax function is used to obtain the predicted emotion recognition result. The cross-entropy loss function is calculated by comparing this result with the actual emotion category label of the sample; A loss function calculation module is used to perform data augmentation on the image and text data obtained in the data extraction module to obtain an image-text fusion vector representation after data augmentation, compare the image-text fusion vector representation with the initial image-text fusion vector representation obtained in the initial image-text fusion vector representation acquisition module, and calculate the loss function. The network parameter optimization module adds the loss functions in the feature alignment module, cross entropy loss calculation module, and loss function calculation module to optimize the network parameters.

Images