Multi-modal emotion detection method based on modal synchronization

Abstract

The invention provides a multi-modal emotion detection method based on modal synchronization. An emotion detection model comprises an image self-attention module used for generating image embedding according to an image vector; the text self-attention module is used for generating text embedding according to the text vector; the image cross-modal conversion module is used for generating an image modal conversion vector according to the image vector and the text vector; the image fusion module is used for generating image single-mode embedding according to the image mode conversion vector and the image embedding; the text cross-modal conversion module is used for generating a text modal conversion vector according to the image vector and the text vector; the text fusion module is used for generating text single-mode embedding according to the text mode conversion vector and the text embedding; the fusion coding module is used for generating a fusion vector according to image single-mode embedding and text single-mode embedding; and the multi-layer sensing module is used for obtaining an emotion detection result according to the fusion vector. In a word, the method can obtain an accurate multi-modal sentiment classification result.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a multimodal emotion detection method based on modal synchronization. Background Art

[0002] With the increasing amount of multimodal data on various social media applications and online shopping platforms, multimodal sentiment detection has become a hot topic in recent years. Compared with traditional text sentiment detection and visual sentiment detection, multimodal sentiment detection not only establishes the correspondence between text entities and visual objects, but also reflects the consistency of different modalities at the sentiment level through underlying and intermediate visual features.

[0003] However, there are two new challenges to achieve accurate multimodal emotion detection.

[0004] First, labeling multimodal data is a time-consuming and labor-intensive task that requires expert evaluation and a complex labeling process, which also results in most existing multimodal sentiment detection datasets having very little labeled data. The limited labeled data not only restricts the training of multimodal sentiment detection models, but also affects the model's ability to generalize to unseen data. However, most existing multimodal sentiment detection methods are trained on small labeled datasets and are unable to utilize the rich unlabeled multimodal data on the Internet to improve the model's sentiment detection capabilities.

[0005] Second, multimodal sentiment detection faces the challenge of modal alignment. The various types of data used for fusion, such as text, images, and sounds, are not only different in format and structure, but also in semantics; and the features extracted from different modalities are also very different. This misalignment phenomenon will cause the model to lose information or mislead training during the fusion process, thus affecting the accuracy of multimodal sentiment detection.

[0006] Therefore, there is an urgent need to propose a more effective multimodal emotion detection method that can not only utilize the unlabeled multimodal image and text data on the Internet, but also achieve multimodal synchronous fusion. Summary of the invention

[0007] The present invention is made to solve the above-mentioned problems, and aims to provide a multimodal emotion detection method based on modal synchronization.

[0008] The present invention provides a multimodal emotion detection method based on modal synchronization, which is used to obtain a corresponding emotion detection result according to a specified picture and a corresponding specified text, and has the following characteristics, including the following steps: step S1, constructing an emotion detection model; step S2, constructing a training data set according to existing image and text data, and training the emotion detection model according to the training data set to obtain a trained emotion detection model; step S3, inputting the specified picture and the specified text into the emotion detection model to obtain an emotion detection result, wherein the emotion detection model includes: an image encoding module, which is used to encode the specified picture to obtain an image vector; an image self-attention module, which is used to perform self-attention processing on the image vector to obtain an image embedding; a text encoding module, which is used to encode the specified text to obtain a text embedding. this vector; a text self-attention module, which is used to perform self-attention processing on the text vector to obtain text embedding; an image cross-modal conversion module, which is used to generate an image modal conversion vector based on the image vector and the text vector; an image fusion module, which is used to splice the image modal conversion vector and the image embedding to obtain an image unimodal embedding; a text cross-modal conversion module, which is used to generate a text modal conversion vector based on the image vector and the text vector; a text fusion module, which is used to splice the text modal conversion vector and the text embedding to obtain a text unimodal embedding; a fusion encoding module, which is used to perform cross-modal attention processing on the image unimodal embedding and the text unimodal embedding to obtain a fusion vector; a multi-layer perception module, including a multi-layer perceptron, is used to process the fusion vector to obtain a sentiment detection result.

[0009] The multimodal emotion detection method based on modal synchronization provided by the present invention may also have the following characteristics: wherein, in the image self-attention module, the calculation expression of image embedding is: Where X v is the image embedding, V is the image vector, is the weight matrix of the query in image mode, is the weight matrix of the key of the image modality, is the weight matrix of the image modality value, d k is the dimension of the key vector. In the text self-attention module, the calculation expression of text embedding is: Where X t is text embedding, T is text vector, is the weight matrix of text-modal queries, is the weight matrix of the key in text mode, A weight matrix for the values ​​of the text modality.

[0010] The multimodal emotion detection method based on modal synchronization provided by the present invention may also have the following characteristics: wherein, in the image cross-modal conversion module, the calculation expression of the image modal conversion vector is: Where X v2t is the image modality conversion vector, T is the text vector, V is the image vector, is the weight matrix of the query in image mode, is the weight matrix of the key of the image modality, is the weight matrix of the image modality value, d k is the dimension of the key vector. In the text cross-modal conversion module, the calculation expression of the text modality conversion vector is: Where X t2v for, is the weight matrix of text-modal queries, is the weight matrix of the key in text mode, A weight matrix for the values ​​of the text modality.

[0011] The multimodal emotion detection method based on modal synchronization provided by the present invention may also have the following characteristics: wherein, in the image fusion module, the calculation expression of the image single modal embedding is: In the formula is the image unimodal embedding, concat is the concatenation operation, X v is the image embedding, X v2t is the image modality conversion vector. In the text fusion module, the calculation expression of text unimodal embedding is: In the formula is the unimodal embedding of text, X t is text embedding, X t2v is the text mode conversion vector.

[0012] The multimodal emotion detection method based on modal synchronization provided by the present invention may also have the following characteristics: wherein, in the fusion coding module, the calculation expression of the fusion vector is: Where X fusion is the fusion vector, is the text unimodal embedding, is the image unimodal embedding, is the weight matrix of the query in text mode, is the weight matrix of the key of the image modality, is the weight matrix of the image modality value, d k is the dimension of the key vector.

[0013] In the multimodal emotion detection method based on modal synchronization provided by the present invention, it can also have the following characteristics: wherein the multilayer perceptron generates a probability vector according to the fusion vector, and the calculation expression of the probability vector is: P = softmax (WX fusion +b), where X fusion is the fusion vector, W is the weight matrix, and b is the bias vector.

[0014] In the multimodal emotion detection method based on modal synchronization provided by the present invention, it can also have the following characteristics: wherein, step S2 includes the following sub-steps: step S2-1, constructing a training data set according to existing image and text data; step S2-2, pre-training the emotion detection model according to the training data set to obtain a pre-trained emotion detection model; step S2-3, fine-tuning the pre-trained emotion detection model according to the training data set to obtain a trained emotion detection model, the training data set includes multiple image-text pairs consisting of training images, corresponding training texts and corresponding true labels, in step S2-2, calculating the overall loss according to the training data set, and updating the parameters of the image encoding module, image self-attention module, text encoding module, text self-attention module, image cross-modal conversion module and text cross-modal conversion module in the emotion detection model according to the overall loss, and in step S2-3, calculating the loss according to the training data set, and updating the parameters of the multi-layer perception module in the pre-trained emotion detection model according to the loss.

[0015] In the multimodal emotion detection method based on modal synchronization provided by the present invention, it can also have the following characteristics: wherein the image encoding module includes an image encoder, the text encoding module includes a text encoder, and the calculation expression of the overall loss is: Loss = Loss itc +Loss itd +Loss mlm +Loss itm , Loss itm =-(y itm logp itm +(1-y itm )log(1-p itm )), where Loss is the overall loss, |queue t | is the length of the text vector queue obtained by the momentum text encoder, |queue v | is the length of the image vector queue obtained by the momentum image encoder, is the image modality conversion vector corresponding to the training image in the i-th image-text pair, is the text vector corresponding to the training text in the i-th image-text pair, is the vector generated by processing the jth training text in the text vector queue by the momentum text encoder, is the text modality conversion vector corresponding to the training text in the i-th image-text pair, is the image vector corresponding to the training image in the i-th image-text pair, is the vector generated by processing the jth training image in the image vector queue through the momentum image encoder, τ is the temperature parameter, P(X v ) is the output of the image encoder processing the training image in the image-text pair, Q(X v ) is the output of the momentum image encoder processing the training image, P(X t ) is the output of the text encoder processing the training text in the image-text pair, Q(X t ) is the output of the momentum text encoder processing the training text, V is the size of the vocabulary, is the one-hot encoding of the i-th word, The probability that the word at the mask position is the i-th word in the vocabulary, y, is predicted by the sentiment detection model itm is the true label corresponding to the image-text pair input into the sentiment detection model, p itm The predicted probability output generated by the sentiment detection model according to the image-text pair, the momentum image encoder has the same structure as the image encoder, and the momentum image encoder performs momentum update according to the parameters of the image encoder. The momentum text encoder has the same structure as the text encoder, and the momentum text encoder performs momentum update according to the parameters of the text encoder.

[0016] In the multimodal emotion detection method based on modal synchronization provided by the present invention, it can also have the following characteristics: wherein the calculation expression of the loss is: Where Loss is the loss, N is the total number of image-text pairs, M is the total number of sentiment categories corresponding to the true labels, and P is i,j The probability that the input i-th image-text pair belongs to the j-th sentiment category predicted by the pre-trained sentiment detection model.

[0017] Functions and Effects of the Invention

[0018] According to the multimodal emotion detection method based on modal synchronization involved in the present invention, on the one hand, the vector of one modality is converted to the semantic space of another modality through the cross-modal conversion module, so that the model can compare vectors of different modalities in the same semantic space; on the other hand, the input is synchronized in dimension through the single-modal fusion module. Therefore, the multimodal emotion detection method based on modal synchronization of the present invention can obtain accurate multimodal emotion classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of a multimodal emotion detection method based on modal synchronization in an embodiment of the present invention;

[0020] Figure 2 is a block diagram of an emotion detection model in an embodiment of the present invention;

[0021] Figure 3 It is a flowchart of emotion detection model training in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments and the accompanying drawings specifically illustrate the multimodal emotion detection method based on modal synchronization of the present invention.

[0023] This embodiment provides a multimodal emotion detection method based on modal synchronization, which is used to obtain a corresponding emotion detection result according to a specified image and a corresponding specified text. In this embodiment, the emotion detection result is one of positive, neutral and negative.

[0024] Figure 1 It is a flow chart of a multimodal emotion detection method based on modal synchronization in an embodiment of the present invention.

[0025] like Figure 1 As shown, the multimodal emotion detection method based on modal synchronization includes the following steps:

[0026] Step S1, constructing an emotion detection model.

[0027] Figure 2 is a block diagram of an emotion detection model in an embodiment of the present invention.

[0028] like Figure 2 As shown, the emotion detection model 100 includes an image encoding module 101, an image self-attention module 102, a text encoding module 103, a text self-attention module 104, an image cross-modal conversion module 105, an image fusion module 106, a text cross-modal conversion module 107, a text fusion module 108, a fusion encoding module 109 and a multi-layer perception module 110.

[0029] The image encoding module 101 includes an image encoder, which is used to encode a specified image to obtain an image vector. In this embodiment, the image encoder is a 12-layer visual transformer, which is initialized by parameters pre-trained on ImageNet-1k.

[0030] The image self-attention module 102 is used to perform self-attention processing on the image vector to obtain image embedding.

[0031] Among them, the calculation expression of image embedding is:

[0032]

[0033] Where X v is the image embedding, V is the image vector, is the weight matrix of the query in image mode, is the weight matrix of the key of the image modality, is the weight matrix of the image modality value, d k is the dimension of the key vector.

[0034] The text encoding module 103 includes a text encoder, which is used to encode the specified text to obtain a text vector. In this embodiment, the text encoder is a pre-trained BERT model.

[0035] The text self-attention module 104 is used to perform self-attention processing on the text vector to obtain text embedding.

[0036] Among them, the calculation expression of text embedding is:

[0037]

[0038] Where X t is text embedding, T is text vector, is the weight matrix of the query in text mode, is the weight matrix of the key in text mode, A weight matrix for the values ​​of the text modality.

[0039] Existing emotion detection methods directly input the output results of the unimodal encoder into the fusion encoder. Due to the structural differences and inconsistent output dimensions between the unimodal encoders, modal synchronization problems may occur during the fusion process. In this embodiment, a cross-modal conversion module, namely an image cross-modal conversion module 105 and a text cross-modal conversion module 107, is introduced to realize the conversion of a vector of one modality to the semantic space of another modality, so that the model can compare vectors of different modalities in the same semantic space.

[0040] The image cross-modal conversion module 105 is used to generate an image modality conversion vector according to the image vector and the text vector.

[0041] Among them, the calculation expression of the image mode conversion vector is:

[0042]

[0043] Where Xv2t is the image modality conversion vector, T is the text vector, V is the image vector, is the weight matrix of the query in image mode, is the weight matrix of the key of the image modality, is the weight matrix of the image modality value, d k is the dimension of the key vector.

[0044] After the cross-modal conversion, the cross-modal conversion embedding and the unimodal embedding are both in the same semantic space. In order to ensure that the two inputs of the multimodal fusion encoder are synchronized in dimension, in this embodiment, the unimodal fusion modules, namely the image fusion module 106 and the text fusion module 108, are constructed according to the dimensional consistency strategy, so as to further synchronize the dimensions of the image and text modalities before fusion.

[0045] The image fusion module 106 is used to concatenate the image modality conversion vector and the image embedding to obtain the image single modality embedding.

[0046] Among them, the calculation expression of image unimodal embedding is:

[0047]

[0048] In the formula is the image unimodal embedding, concat is the concatenation operation, X v is the image embedding, X v2t is the image mode conversion vector.

[0049] The text cross-modal conversion module 107 is used to generate a text modality conversion vector according to the image vector and the text vector.

[0050] Among them, the calculation expression of the text mode conversion vector is:

[0051]

[0052] Where X t2v for, is the weight matrix of text-modal queries, is the weight matrix of the key in text mode, A weight matrix for the values ​​of the text modality.

[0053] The text fusion module 108 is used to concatenate the text modality conversion vector and the text embedding to obtain the text unimodal embedding.

[0054] Among them, the calculation expression of text unimodal embedding is:

[0055]

[0056] In the formula is the unimodal embedding of text, X t is text embedding, X t2v is the text mode conversion vector.

[0057] The fusion encoding module 109 is used to perform cross-modal attention processing on the image unimodal embedding and the text unimodal embedding to obtain a fusion vector.

[0058] Among them, the calculation expression of the fusion vector is:

[0059]

[0060] Where X fusion is the fusion vector, is the text unimodal embedding, is the image unimodal embedding, is the weight matrix of the query in text mode, is the weight matrix of the key of the image modality, is the weight matrix of the image modality value, d k is the dimension of the key vector.

[0061] The multi-layer perception module 110 includes a multi-layer perceptron, which is used to process the fusion vector to obtain a sentiment detection result.

[0062] Among them, the multilayer perceptron generates a probability vector based on the fusion vector, and the calculation expression of the probability vector is:

[0063] P = softmax(WX fusion +b),

[0064] Where X fusion is the fusion vector, W is the weight matrix, and b is the bias vector.

[0065] Step S2, constructing a training data set according to the existing graphic data, and training the emotion detection model 100 according to the training data set to obtain a trained emotion detection model 100.

[0066] Figure 3 It is a flowchart of emotion detection model training in an embodiment of the present invention.

[0067] like Figure 3 As shown, step S2 includes the following sub-steps:

[0068] Step S2-1, constructing a training data set based on existing image and text data. The training data set includes a plurality of image-text pairs consisting of training images, corresponding training texts and corresponding true labels.

[0069] Step S2-2, pre-training the emotion detection model according to the training data set to obtain a pre-trained emotion detection model.

[0070] Among them, the overall loss is calculated according to the training data set in pre-training, and the parameters of the image encoding module, image self-attention module, text encoding module, text self-attention module, image cross-modal conversion module and text cross-modal conversion module in the emotion detection model are updated according to the overall loss.

[0071] In step S2-2 of this embodiment, the training data set is an image-text pair that does not contain an emotion label. In order to make full use of the training data set constructed by the image-text pair that does not contain an emotion label and to explore the connection between different modalities, this embodiment designs a multimodal synchronous contrastive learning task for the self-supervised training of the model. Unlike other contrastive learning tasks, the multimodal synchronous contrastive learning task of this embodiment generates a cross-modal conversion vector, so that the contrast loss can be calculated in a unified feature space, thereby achieving the synchronization of modalities.

[0072] Since the distance between the vectors of the same set of training images and training texts in the feature space is closer, the vectors of the two modes of each sample are set as positive samples of each other during the training process, and the output vectors obtained from all other training images and training texts in the same batch through the cross-modal conversion module are used as negative samples. Using only samples from a single batch for training will be affected by too few samples, and using samples from other batches as negative samples will affect the consistency of sample encoding. Therefore, this embodiment adds two new encoders in parallel, namely the momentum image encoder and the momentum text encoder.

[0073] The momentum image encoder has the same structure as the image encoder, and the momentum image encoder updates the momentum according to the parameters of the image encoder. The momentum text encoder has the same structure as the text encoder, and the momentum text encoder updates the momentum according to the parameters of the text encoder. The specific method of momentum update in this embodiment is to update the parameters of the corresponding unimodal encoder, i.e., the text encoder or the image encoder, slowly and indirectly, so as to ensure the consistency of the encoding.

[0074] In addition, pre-training is used to optimize the model's handling of language masking tasks and image-text matching tasks. For the language masking task, some text tags are randomly masked, and the remaining contextual text and image information are used to predict the masked parts. For the image-text matching task, the goal is to predict whether there is a matching relationship between the image and the text through the fusion vector. This embodiment uses the output embedding of the multimodal fusion encoder as a joint representation of the image-text pair, and attaches a fully connected layer as a classifier for the matching relationship.

[0075] Then, the calculation expression of the overall loss of pre-training is:

[0076] Loss=Loss itc +Loss itd +Loss mlm +Loss itm ,

[0077]

[0078] Loss itm =-(y itml ogp itm +(1-y itm )log(1-p itm )),

[0079] Where Loss is the overall loss, |queue t | is the length of the text vector queue obtained by the momentum text encoder, |queue v | is the length of the image vector queue obtained by the momentum image encoder, is the image modality conversion vector corresponding to the training image in the i-th image-text pair, is the text vector corresponding to the training text in the i-th image-text pair, is the vector generated by processing the jth training text in the text vector queue by the momentum text encoder, is the text modality conversion vector corresponding to the training text in the i-th image-text pair, is the image vector corresponding to the training image in the i-th image-text pair, is the vector generated by processing the jth training image in the image vector queue through the momentum image encoder, τ is the temperature parameter, P(X v ) is the output of the image encoder processing the training image in the image-text pair, Q(X v ) is the output of the momentum image encoder processing the training image, P(X t ) is the output of the text encoder processing the training text in the image-text pair, Q(X t ) is the output of the momentum text encoder processing the training text, V is the size of the vocabulary, is the one-hot encoding of the i-th word, The probability that the word at the mask position is the i-th word in the vocabulary, y, is predicted by the sentiment detection model itm is the true label corresponding to the image-text pair input into the sentiment detection model, p itm It is the predicted probability output generated by the emotion detection model based on the image-text pair. In this embodiment, the temperature parameter τ is used to control the distribution of the loss function. The larger τ is, the smoother the distribution of the loss is.

[0080] Step S2-3, fine-tuning the pre-trained emotion detection model 100 according to the training data set to obtain the trained emotion detection model 100.

[0081] In the fine-tuning, the loss is calculated according to the training data set, and the parameters of the multi-layer perception module 110 in the pre-trained emotion detection model are updated according to the loss. In this embodiment, when the multi-layer perception module 110 is fine-tuned, the initial parameters of the multi-layer perception module 110 are randomly initialized parameters.

[0082] The loss calculation expression is:

[0083]

[0084] Where Loss is the loss, N is the total number of image-text pairs, M is the total number of sentiment categories corresponding to the true labels, and P is i,j The probability that the input i-th image-text pair belongs to the j-th sentiment category predicted by the pre-trained sentiment detection model.

[0085] Step S3: input the specified picture and the specified text into the emotion detection model 100 to obtain the emotion detection result.

[0086] In this embodiment, the performance of the emotion detection model 100, namely Ours model, is compared with the existing multimodal emotion models with good performance, namely VAuLT model and ALBEF model. First, the emotion detection model 100 is pre-trained based on the existing MSCOCO dataset containing more than 100,000 pairs of unlabeled image-text pairs, and then the Ours model, VAuLT model and ALBEF model are fine-tuned on the existing MVSA-Single dataset, MVSA-Multiple dataset and RU-Senti dataset containing multiple pairs of labeled image-text pairs, respectively, and then the classification accuracy Acc and F1 score of each model for the three classifications of positive, neutral and negative emotions on the above three datasets are obtained as shown in the following table:

[0087]

[0088] In the above table, the first column is each model, the second and third columns are the Acc and F1 scores of each model on the MVSA-Single dataset, the fourth and fifth columns are the Acc and F1 scores of each model on the MVSA-Multiple dataset, and the sixth and seventh columns are the Acc and F1 scores of each model on the RU-Senti dataset. For example, the cell in the second row and second column indicates that the classification accuracy Acc of the VAuLT model on the MVSA-Single dataset is 75.60%. It can be seen that the emotion detection model 100 has better multimodal emotion classification performance than the two existing models.

[0089] In this embodiment, an ablation experiment is further conducted on the RU-Senti dataset to verify the effects of pre-training, cross-modal conversion module, and Loss. itc Recorded as ITC loss, Loss itd Recorded as ITD loss, Loss mlm Recorded as MLM loss and Loss itm The impact of ITM loss on the performance of emotion detection model 100 is shown in the following table:

[0090] Acc F1 score Remove pre-training 81.46％ 0.7855 Remove cross-modal conversion module 82.87％ 0.7989 Remove ITC loss 82.52％ 0.8007 Removal of ITD loss 82.26％ 0.7933 Remove MLM losses 82.98％ 0.8022 Removing ITM loss 82.68％ 0.7971 Complete emotion detection model 84.98％ 0.8401

[0091] The first column in the table above shows the various ablation models, and the second and third columns show the corresponding Acc and F1 scores of each model on the RU-Senti dataset. For example, the cell in the second column of the eighth row shows that the classification accuracy of the complete emotion detection model 100 obtained by the multimodal emotion detection method based on modal synchronization is 84.98%. It can be seen that the pre-training, cross-modal conversion module, and Loss in this embodiment are itc 、Loss itd 、Loss mlm and Loss itm All of them can improve the multimodal sentiment classification performance of the model, and the complete sentiment detection model 100 obtained by combining the above features has the best multimodal sentiment classification performance.

[0092] Functions and Effects of the Embodiments

[0093] According to the multimodal emotion detection method based on modal synchronization involved in this embodiment, on the one hand, the vector of one modality is converted to the semantic space of another modality through the cross-modal conversion module, so that the model can compare vectors of different modalities in the same semantic space; on the other hand, the input is synchronized in dimension through the single-modal fusion module. In short, this method can obtain accurate multimodal emotion classification results.

[0094] Furthermore, a momentum image encoder and a momentum text encoder are set in pre-training, and the vectors of the two modes of the same sample are used as positive samples of each other, and other samples in the training data are used as negative samples. This can improve the training sample data in unlabeled training without affecting the consistency of sample encoding, thereby obtaining a model with better performance.

[0095] Furthermore, language masking tasks and image-text matching tasks and their corresponding loss functions are set in the pre-training, so that the pre-trained model can cope with language masking tasks and image-text matching tasks, further improving the model performance.

[0096] Furthermore, during fine-tuning, the parameters of the multi-layer perception module are optimized through labeled training data, which reduces the demand for labeled training data and improves the prediction accuracy of the model.

[0097] Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A multimodal emotion detection method based on modal synchronization, used to obtain a corresponding emotion detection result according to a specified picture and a corresponding specified text, characterized in that: The following steps are involved: Step S1, constructing an emotion detection model; Step S2, constructing a training data set according to the existing graphic data, and training the emotion detection model according to the training data set to obtain a trained emotion detection model; Step S3, inputting the specified picture and the specified text into the emotion detection model to obtain the emotion detection result, Wherein, the emotion detection model includes: An image encoding module, used for encoding the specified image to obtain an image vector; An image self-attention module, used to perform self-attention processing on the image vector to obtain image embedding; A text encoding module, used for encoding the specified text to obtain a text vector; A text self-attention module, used to perform self-attention processing on the text vector to obtain text embedding; An image cross-modal conversion module, used to generate an image modality conversion vector according to the image vector and the text vector; An image fusion module, used for concatenating the image modality conversion vector and the image embedding to obtain an image single modality embedding; A text cross-modal conversion module, used to generate a text modality conversion vector according to the image vector and the text vector; A text fusion module, used for concatenating the text modality conversion vector and the text embedding to obtain a text unimodal embedding; A fusion encoding module, used for performing cross-modal attention processing on the image unimodal embedding and the text unimodal embedding to obtain a fusion vector; The multi-layer perception module includes a multi-layer perceptron, which is used to process the fusion vector to obtain the emotion detection result.

2. The multimodal emotion detection method based on modal synchronization according to claim 1, characterized in that: in, In the image self-attention module, the computational expression for image embedding is: Where X v is the image embedding, V is the image vector, is the weight matrix of the query in image mode, is the weight matrix of the key of the image modality, is the weight matrix of the image modality value, d k is the dimension of the key vector, In the text self-attention module, the calculation expression of the text embedding is: Where X t is the text embedding, T is the text vector, is the weight matrix of text-modal queries, is the weight matrix of the key in text mode, A weight matrix for the values ​​of the text modality.

3. The multimodal emotion detection method based on modal synchronization according to claim 1, characterized in that: in, In the image cross-modal conversion module, the calculation expression of the image modality conversion vector is: Where X v2t is the image modality conversion vector, T is the text vector, V is the image vector, is the weight matrix of the query in image mode, is the weight matrix of the key of the image modality, is the weight matrix of the image modality value, d k is the dimension of the key vector, In the text cross-modal conversion module, the calculation expression of the text modality conversion vector is: Where X t2v for, is the weight matrix of text-modal queries, is the weight matrix of the key in text mode, A weight matrix for the values ​​of the text modality.

4. The multimodal emotion detection method based on modal synchronization according to claim 1, characterized in that: in, In the image fusion module, the calculation expression of the image single modality embedding is: In the formula is the unimodal embedding of the image, concat is the concatenation operation, X v For the image embedding, X v2t is the image modality conversion vector, In the text fusion module, the calculation expression of the text unimodal embedding is: In the formula is the unimodal embedding of the text, X t For the text embedding, X t2v is the text mode conversion vector.

5. The multimodal emotion detection method based on modal synchronization according to claim 1, characterized in that: in, In the fusion coding module, the calculation expression of the fusion vector is: Where X fusion is the fusion vector, is the unimodal embedding of the text, is the unimodal embedding of the image, is the weight matrix of text-modal queries, is the weight matrix of the key of the image modality, is the weight matrix of the image modality value, d k is the dimension of the key vector.

6. The multimodal emotion detection method based on modal synchronization according to claim 1, characterized in that: in, The multilayer perceptron generates a probability vector according to the fusion vector, The calculation expression of the probability vector is: P=softmax(WX fusion +b), Where X fusion is the fusion vector, W is the weight matrix, and b is the bias vector.

7. The multimodal emotion detection method based on modality synchronization according to claim 1, Features: in, The step S2 comprises the following sub-steps: Step S2-1, constructing a training data set based on existing graphic and text data; Step S2-2, pre-training the emotion detection model according to the training data set to obtain a pre-trained emotion detection model; Step S2-3, fine-tuning the pre-trained emotion detection model according to the training data set to obtain the trained emotion detection model, The training data set includes a plurality of image-text pairs consisting of training images, corresponding training texts and corresponding true labels. In the step S2-2, the overall loss is calculated according to the training data set, and the parameters of the image encoding module, the image self-attention module, the text encoding module, the text self-attention module, the image cross-modal conversion module and the text cross-modal conversion module in the emotion detection model are updated according to the overall loss. In step S2-3, the loss is calculated based on the training data set, and the parameters of the multi-layer perception module in the pre-trained emotion detection model are updated based on the loss.

8. The multimodal emotion detection method based on modal synchronization according to claim 7, characterized in that: in, The image encoding module includes an image encoder, The text encoding module includes a text encoder, The calculation expression of the overall loss is: Loss=Loss itc +Loss itd +Loss mlm +Loss itm , Loss itm =-(y itm logp itm +(1-y itm )log(1-p itm )), Where Loss is the overall loss, |queue t | is the length of the text vector queue obtained by the momentum text encoder, |queue v | is the length of the image vector queue obtained by the momentum image encoder, is the image modality conversion vector corresponding to the training image in the i-th image-text pair, is the text vector corresponding to the training text in the i-th image-text pair, is the vector generated by processing the jth training text in the text vector queue by the momentum text encoder, is the text modality conversion vector corresponding to the training text in the i-th image-text pair, is the image vector corresponding to the training image in the i-th image-text pair, is the vector generated by processing the jth training image in the image vector queue through the momentum image encoder, τ is the temperature parameter, P(X v ) is the output obtained by the image encoder processing the training image in the image-text pair, Q(X v ) is the output obtained by the momentum image encoder processing the training image, P(X t ) is the output obtained by the text encoder processing the training text in the image-text pair, Q(X t ) is the output obtained by the momentum text encoder processing the training text, V is the size of the vocabulary, is the one-hot encoding of the i-th word, The probability that the word at the mask position predicted by the emotion detection model is the i-th word in the vocabulary, yi t m is the true label corresponding to the image-text pair input into the emotion detection model, p itm is the predicted probability output generated by the emotion detection model according to the image-text pair, The momentum image encoder has the same structure as the image encoder. The momentum image encoder performs momentum update according to the parameters of the image encoder, The momentum text encoder has the same structure as the text encoder, The momentum text encoder performs momentum update according to the parameters of the text encoder.

9. The multimodal emotion detection method based on modal synchronization according to claim 7, characterized in that: in, The loss calculation expression is: Where Loss is the loss, N is the total number of image-text pairs, M is the total number of sentiment categories corresponding to the true label, and P i，j The probability that the input i-th image-text pair belongs to the j-th emotion category predicted by the pre-trained emotion detection model.

Images