Modality-synchronous multi-modal sentiment detection method

Abstract

The application provides a multi-modal emotion detection method based on modal synchronization, wherein the emotion detection model comprises: an image self-attention module for generating image embedding according to an image vector; a text self-attention module for generating text embedding according to a text vector; an image cross-modal conversion module for generating an image modal conversion vector according to the image vector and the text vector; an image fusion module for generating an image single-modal embedding according to the image modal conversion vector and the image embedding; a text cross-modal conversion module for generating a text modal conversion vector according to the image vector and the text vector; a text fusion module for generating a text single-modal embedding according to the text modal conversion vector and the text embedding; a fusion encoding module for generating a fusion vector according to the image single-modal embedding and the text single-modal embedding; and a multi-layer perception module for obtaining an emotion detection result according to the fusion vector. In summary, the method can obtain an accurate multi-modal emotion classification result.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and specifically to a multimodal emotion detection method based on modal synchronization. Background Technology

[0002] With the increasing volume of multimodal data on various social media applications and online shopping platforms, multimodal sentiment detection has become a research hotspot in recent years. Compared with traditional text sentiment detection and visual sentiment detection, multimodal sentiment detection not only establishes the correspondence between text entities and visual objects, but also reflects the consistency of different modalities at the sentiment level through underlying and intermediate visual features.

[0003] However, there are two new challenges to achieving accurate multimodal emotion detection.

[0004] First, labeling multimodal data is a time-consuming and labor-intensive task, requiring expert evaluation and a complex labeling process. This results in most existing multimodal sentiment detection datasets having very limited labeled data. The limited labeled data not only restricts the training of multimodal sentiment detection models but also affects their ability to generalize to unseen data. However, most existing multimodal sentiment detection methods are trained on small labeled datasets, unable to leverage the abundant unlabeled multimodal data available online to improve their sentiment detection capabilities.

[0005] Second, multimodal sentiment detection faces the challenge of modality alignment. The various types of data used for fusion, such as text, images, and audio, differ not only in format and structure but also in semantics; furthermore, the features extracted from different modalities vary significantly. This misalignment can lead to information loss or misleading training during the fusion process, thus affecting the accuracy of multimodal sentiment detection.

[0006] Therefore, there is an urgent need to propose a more effective multimodal sentiment detection method that can utilize unlabeled multimodal text and image data from the Internet and achieve simultaneous fusion of multimodal data. Summary of the Invention

[0007] This invention is made to solve the above-mentioned problems, and aims to provide a multimodal emotion detection method based on modal synchronization.

[0008] This invention provides a multimodal sentiment detection method based on modal synchronization, used to obtain corresponding sentiment detection results based on a specified image and corresponding specified text. It includes the following steps: Step S1, constructing a sentiment detection model; Step S2, constructing a training dataset based on existing image and text data, and training the sentiment detection model using the training dataset to obtain a trained sentiment detection model; Step S3, inputting the specified image and specified text into the sentiment detection model to obtain the sentiment detection result. The sentiment detection model includes: an image encoding module for encoding the specified image to obtain an image vector; an image self-attention module for performing self-attention processing on the image vector to obtain an image embedding; and a text encoding module for encoding the specified text to obtain a text embedding. The module consists of: a text self-attention module for processing the text vector to obtain a text embedding; an image cross-modal transformation module for generating an image modality transformation vector based on the image vector and the text vector; an image fusion module for concatenating the image modality transformation vector and the image embedding to obtain an image single-modal embedding; a text cross-modal transformation module for generating a text modality transformation vector based on the image vector and the text vector; a text fusion module for concatenating the text modality transformation vector and the text embedding to obtain a text single-modal embedding; a fusion encoding module for performing cross-modal attention processing on the image single-modal embedding and the text single-modal embedding to obtain a fused vector; and a multilayer perceptron module, including a multilayer perceptron, for processing the fused vector to obtain the emotion detection result.

[0009] The multimodal emotion detection method based on modal synchronization provided in this invention may also have the following feature: In the image self-attention module, the calculation expression for image embedding is: In the formula X v For image embedding, V is the image vector. The weight matrix for the query of the image modality. The weight matrix of the keys for the image modality. Let d be the weight matrix of the image modal values. k Let be the dimension of the key vector. In the text self-attention module, the calculation expression for text embedding is: In the formula X t For text embedding, T is the text vector. The weight matrix for text-modal queries. This is the weight matrix of the keys in the text modality. This is the weight matrix of the values ​​of the text modality.

[0010] The multimodal emotion detection method based on modal synchronization provided by this invention may also have the following feature: In the image cross-modal conversion module, the calculation expression for the image modal conversion vector is: In the formula X v2t Let T be the image modality transformation vector, and V be the text vector and V be the image vector. The weight matrix for the query of the image modality. The weight matrix of the keys for the image modality. Let d be the weight matrix of the image modal values. k Let be the dimension of the key vector. In the text cross-modal conversion module, the expression for calculating the text modality conversion vector is: In the formula X t2v for, The weight matrix for text-modal queries. This is the weight matrix of the keys in the text modality. This is the weight matrix of the values ​​of the text modality.

[0011] The multimodal emotion detection method based on modal synchronization provided by this invention may also have the following feature: In the image fusion module, the calculation expression for image unimodal embedding is: In the formula For image unimodal embedding, concat is the concatenation operation, X v For image embedding, X v2t Given the image modality transformation vector, the calculation expression for text unimodal embedding in the text fusion module is: In the formula X is a text-based unimodal embedding. t For text embedding, X t2v This is a text modality transformation vector.

[0012] The multimodal emotion detection method based on modal synchronization provided by this invention may also have the following feature: wherein, in the fusion encoding module, the calculation expression for the fusion vector is: In the formula X fusion For the fusion vector, For text-based unimodal embedding, For image single-modal embedding, The weight matrix for text-modal queries. The weight matrix of the keys for the image modality. Let d be the weight matrix of the image modal values. k is the dimension of the key vector.

[0013] The multimodal emotion detection method based on modal synchronization provided in this invention may also have the following feature: The multilayer perceptron generates a probability vector based on the fusion vector, and the calculation expression for the probability vector is: P = softmax(WX) fusion +b), where X fusion Let W be the fusion vector, W be the weight matrix, and b be the bias vector.

[0014] The multimodal emotion detection method based on modal synchronization provided by this invention may also have the following features: Step S2 includes the following sub-steps: Step S2-1, constructing a training dataset based on existing image and text data; Step S2-2, pre-training the emotion detection model based on the training dataset to obtain a pre-trained emotion detection model; Step S2-3, fine-tuning the pre-trained emotion detection model based on the training dataset to obtain a trained emotion detection model. The training dataset includes multiple image-text pairs consisting of training images, corresponding training text, and corresponding real labels. In Step S2-2, the overall loss is calculated based on the training dataset, and the parameters of the image encoding module, image self-attention module, text encoding module, text self-attention module, image cross-modal conversion module, and text cross-modal conversion module in the emotion detection model are updated based on the overall loss. In Step S2-3, the loss is calculated based on the training dataset, and the parameters of the multilayer perception module in the pre-trained emotion detection model are updated based on the loss.

[0015] The multimodal emotion detection method based on modal synchronization provided in this invention may also have the following features: the image encoding module includes an image encoder, the text encoding module includes a text encoder, and the overall loss calculation expression is: Loss = Loss itc +Loss itd +Loss mlm +Loss itm , Loss itm =-(y itm logp itm +(1-y itm log(1-p) itm In the formula, Loss is the total loss, |queue t | represents the length of the queue of text vectors obtained by the momentum text encoder, |queue v | represents the length of the queue of image vectors obtained by the momentum image encoder. Let be the image mode transformation vector corresponding to the training image in the i-th image-text pair. Let be the text vector corresponding to the training text in the i-th image-text pair. This is the vector generated after processing the j-th training text in the text vector queue using a momentum text encoder. Let i be the text modality transformation vector corresponding to the training text in the i-th image-text pair. Let be the image vector corresponding to the training image in the i-th image-text pair. P(X) is the vector generated by processing the j-th training image in the image vector queue using a momentum image encoder, where τ is the temperature parameter. v Q(X) is the output obtained by the image encoder processing the training images in the image-text pair. v P(X) is the output obtained by the momentum image encoder after processing the training image. t Q(X) is the output obtained by the text encoder processing the training text in the image-text pair. t The output of the momentum text encoder is the training text, and V is the size of the vocabulary. For the one-hot encoding of the i-th word, To predict the probability that the word at the mask position in a sentiment detection model is the i-th word in the vocabulary, y itm To input the image-text pairs corresponding to the ground truth labels into the sentiment detection model, p itm The emotion detection model generates predicted probability outputs based on image-text pairs. The momentum image encoder has the same structure as the image encoder and updates momentum based on the parameters of the image encoder. The momentum text encoder has the same structure as the text encoder and updates momentum based on the parameters of the text encoder.

[0016] The multimodal emotion detection method based on modal synchronization provided in this invention may also have the following feature: where the loss calculation expression is: In the formula, Loss represents the loss, N is the total number of image-text pairs, M is the total number of sentiment categories corresponding to the true labels, and P... i,j This is the probability predicted by a pre-trained sentiment detection model that the i-th image-text pair of the input belongs to the j-th sentiment category.

[0017] The role and effect of invention

[0018] According to the modality synchronization-based multimodal sentiment detection method of the present invention, on the one hand, a cross-modal transformation module transforms a vector from one modality to the semantic space of another modality, allowing the model to compare vectors from different modalities within the same semantic space; on the other hand, a single-modal fusion module synchronizes the input in dimensions. Therefore, the modality synchronization-based multimodal sentiment detection method of the present invention can obtain accurate multimodal sentiment classification results. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the multimodal emotion detection method based on modal synchronization in an embodiment of the present invention;

[0020] Figure 2 This is a block diagram of the emotion detection model in an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of the training process of the emotion detection model in an embodiment of the present invention. Detailed Implementation

[0022] To make the technical means, creative features, objectives and effects of this invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate the multimodal emotion detection method based on modal synchronization of this invention.

[0023] This embodiment provides a multimodal sentiment detection method based on modal synchronization, used to obtain corresponding sentiment detection results based on a specified image and corresponding specified text. In this embodiment, the sentiment detection result is one of three: Positive, Neutral, or Negative.

[0024] Figure 1 This is a flowchart illustrating the multimodal emotion detection method based on modal synchronization in an embodiment of the present invention.

[0025] like Figure 1 As shown, the multimodal emotion detection method based on modal synchronization includes the following steps:

[0026] Step S1: Construct an emotion detection model.

[0027] Figure 2 This is a block diagram of the emotion detection model in an embodiment of the present invention.

[0028] like Figure 2 As shown, the emotion detection model 100 includes an image encoding module 101, an image self-attention module 102, a text encoding module 103, a text self-attention module 104, an image cross-modal conversion module 105, an image fusion module 106, a text cross-modal conversion module 107, a text fusion module 108, a fusion encoding module 109, and a multilayer perception module 110.

[0029] The image encoding module 101 includes an image encoder for encoding a specified image to obtain an image vector. In this embodiment, the image encoder is a 12-layer visual transformer, which is initialized using parameters pre-trained on ImageNet-1k.

[0030] The image self-attention module 102 is used to perform self-attention processing on the image vector to obtain the image embedding.

[0031] The calculation expression for image embedding is as follows:

[0032]

[0033] In the formula X v For image embedding, V is the image vector. The weight matrix for the query of the image modality. The weight matrix of the keys for the image modality. Let d be the weight matrix of the image modal values. k is the dimension of the key vector.

[0034] The text encoding module 103 includes a text encoder, used to encode specified text to obtain text vectors. In this embodiment, the text encoder is a pre-trained BERT model.

[0035] The text self-attention module 104 is used to perform self-attention processing on the text vector to obtain the text embedding.

[0036] The calculation expression for text embedding is as follows:

[0037]

[0038] In the formula X t For text embedding, T is the text vector. The weight matrix for text-modal queries. This is the weight matrix of the keys in the text modality. This is the weight matrix of the values ​​of the text modality.

[0039] Existing sentiment detection methods directly input the output of a single-modal encoder into a fusion encoder. Due to structural differences and inconsistent output dimensions between single-modal encoders, modal synchronization problems may occur during the fusion process. This embodiment introduces cross-modal conversion modules, namely image cross-modal conversion module 105 and text cross-modal conversion module 107, which can convert vectors from one modality to the semantic space of another modality, allowing the model to compare vectors from different modalities in the same semantic space.

[0040] The image cross-modal conversion module 105 is used to generate an image modality conversion vector based on the image vector and the text vector.

[0041] The expression for calculating the image mode transformation vector is as follows:

[0042]

[0043] In the formula Xv2t Let T be the image modality transformation vector, and V be the text vector and V be the image vector. The weight matrix for the query of the image modality. The weight matrix of the keys for the image modality. Let d be the weight matrix of the image modal values. k is the dimension of the key vector.

[0044] After cross-modal transformation, the cross-modal embedding and the unimodal embedding reside in the same semantic space. To ensure that the two inputs of the multimodal fusion encoder are dimensionally synchronized, this embodiment constructs a unimodal fusion module, namely image fusion module 106 and text fusion module 108, based on a dimensionality consistency strategy, thereby achieving further synchronization of the dimensions of the image and text modalities before fusion.

[0045] The image fusion module 106 is used to concatenate the image modality transformation vector and the image embedding to obtain the image single modality embedding.

[0046] The calculation expression for image unimodal embedding is as follows:

[0047]

[0048] In the formula For image unimodal embedding, concat is the concatenation operation, X v For image embedding, X v2t This is the image mode transformation vector.

[0049] The text cross-modal conversion module 107 is used to generate a text modality conversion vector based on the image vector and the text vector.

[0050] The expression for calculating the text modality transition vector is as follows:

[0051]

[0052] In the formula X t2v for, The weight matrix for text-modal queries. This is the weight matrix of the keys in the text modality. This is the weight matrix of the values ​​of the text modality.

[0053] The text fusion module 108 is used to concatenate the text modality transformation vector and the text embedding to obtain the text single modality embedding.

[0054] The calculation expression for text unimodal embedding is as follows:

[0055]

[0056] In the formula X is a text-based unimodal embedding. t For text embedding, X t2v This is a text modality transformation vector.

[0057] The fusion coding module 109 is used to perform cross-modal attention processing on image unimodal embedding and text unimodal embedding to obtain a fusion vector.

[0058] The expression for calculating the fusion vector is as follows:

[0059]

[0060] In the formula X fusion For the fusion vector, For text-based unimodal embedding, For image single-modal embedding, The weight matrix for text-modal queries. The weight matrix of the keys for the image modality. Let d be the weight matrix of the image modal values. k is the dimension of the key vector.

[0061] The multilayer perception module 110 includes a multilayer perceptron, which is used to process the fusion vector to obtain the emotion detection result.

[0062] The multilayer perceptron generates a probability vector based on the fused vector, and the expression for calculating the probability vector is as follows:

[0063] P = softmax(WX) fusion +b),

[0064] In the formula X fusion Let W be the fusion vector, W be the weight matrix, and b be the bias vector.

[0065] Step S2: Construct a training dataset based on existing image and text data, and train the sentiment detection model 100 based on the training dataset to obtain the trained sentiment detection model 100.

[0066] Figure 3 This is a schematic diagram of the training process of the emotion detection model in an embodiment of the present invention.

[0067] like Figure 3 As shown, step S2 includes the following sub-steps:

[0068] Step S2-1: Construct a training dataset based on existing image and text data. The training dataset includes multiple image-text pairs consisting of training images, corresponding training text, and corresponding ground truth labels.

[0069] Step S2-2: Pre-train the sentiment detection model based on the training dataset to obtain the pre-trained sentiment detection model.

[0070] In the pre-training process, the overall loss is calculated based on the training dataset, and the parameters of the image encoding module, image self-attention module, text encoding module, text self-attention module, image cross-modal conversion module, and text cross-modal conversion module in the emotion detection model are updated based on the overall loss.

[0071] In step S2-2 of this embodiment, the training dataset consists of image-text pairs without sentiment labels. To fully utilize the training dataset constructed from these image-text pairs and to uncover the connections between different modalities, this embodiment designs a multimodal synchronous contrastive learning task for the model's self-supervised training. Unlike other contrastive learning tasks, this embodiment's multimodal synchronous contrastive learning task generates cross-modal transformation vectors, enabling the contrastive loss to be calculated within a unified feature space, thus achieving modal synchronization.

[0072] Since the distance between training images and training text vectors in the same set is closer in the feature space, during training, the vectors of the two modalities of each sample are considered positive samples, while the output vectors obtained from all other training images and training texts in the same batch through the cross-modal conversion module are considered negative samples. Training using only samples from a single batch would be ineffective due to insufficient samples, while using samples from other batches as negative samples would affect the consistency of sample encoding. Therefore, this embodiment adds two new encoders in parallel: a momentum image encoder and a momentum text encoder.

[0073] The momentum image encoder has the same structure as the image encoder, and it updates momentum based on the parameters of the image encoder. Similarly, the momentum text encoder has the same structure as the text encoder, and it updates momentum based on the parameters of the text encoder. In this embodiment, the momentum update is performed slowly and indirectly based on the parameters of the corresponding single-modal encoder, i.e., the text encoder or the image encoder, thereby ensuring encoding consistency.

[0074] Furthermore, pre-training is used to optimize the model's handling of language masking and image-text matching tasks. For language masking, some text markers are randomly masked, and the remaining contextual text and image information are used to predict the masked portion. For image-text matching, the goal is to predict whether a matching relationship exists between the image and text through a fused vector. This embodiment uses the output embedding of a multimodal fusion encoder as the joint representation of the image-text pair, and adds a fully connected layer as a classifier for the matching relationship.

[0075] Therefore, the expression for calculating the overall loss during pre-training is:

[0076] Loss = Loss itc +Loss itd +Loss mlm +Loss itm ,

[0077]

[0078] Loss itm =-(y itml OGP itm +(1-y itm log(1-p) itm )),

[0079] In the formula, Loss represents the total loss, |queue t | represents the length of the queue of text vectors obtained by the momentum text encoder, |queue v | represents the length of the queue of image vectors obtained by the momentum image encoder. Let be the image mode transformation vector corresponding to the training image in the i-th image-text pair. Let be the text vector corresponding to the training text in the i-th image-text pair. This is the vector generated after processing the j-th training text in the text vector queue using a momentum text encoder. Let i be the text modality transformation vector corresponding to the training text in the i-th image-text pair. Let be the image vector corresponding to the training image in the i-th image-text pair. P(X) is the vector generated by processing the j-th training image in the image vector queue using a momentum image encoder, where τ is the temperature parameter. v Q(X) is the output obtained by the image encoder processing the training images in the image-text pair. v P(X) is the output obtained by the momentum image encoder after processing the training image. t Q(X) is the output obtained by the text encoder processing the training text in the image-text pair. t The output of the momentum text encoder is the training text, and V is the size of the vocabulary. For the one-hot encoding of the i-th word, To predict the probability that the word at the mask position in a sentiment detection model is the i-th word in the vocabulary, y itm To input the image-text pairs corresponding to the ground truth labels into the sentiment detection model, p itm This is the predicted probability output generated by the emotion detection model based on image-text pairs. In this embodiment, the temperature parameter τ is used to control the distribution of the loss function; the larger τ is, the smoother the distribution of the loss.

[0080] Step S2-3: Fine-tune the pre-trained sentiment detection model 100 based on the training dataset to obtain the trained sentiment detection model 100.

[0081] In the fine-tuning process, the loss is calculated based on the training dataset, and the parameters of the multilayer perceptron module 110 in the pre-trained sentiment detection model are updated based on the loss. In this embodiment, when fine-tuning the multilayer perceptron module 110, the initial parameters of the multilayer perceptron module 110 are randomly initialized.

[0082] The expression for calculating the loss is:

[0083]

[0084] In the formula, Loss represents the loss, N is the total number of image-text pairs, M is the total number of sentiment categories corresponding to the true labels, and P... i,j This is the probability predicted by a pre-trained sentiment detection model that the i-th image-text pair of the input belongs to the j-th sentiment category.

[0085] Step S3: Input the specified image and specified text into the sentiment detection model 100 to obtain the sentiment detection result.

[0086] In this embodiment, the sentiment detection model 100, namely the Ours model, is compared with the existing high-performing multimodal sentiment models, namely the VAuLT model and the ALBEF model. First, the sentiment detection model 100 is pre-trained on the existing MSCOCO dataset, which contains over 100,000 unlabeled image-text pairs. Then, the Ours model, VAuLT model, and ALBEF model are fine-tuned sequentially on the existing MVSA-Single dataset, MVSA-Multiple dataset, and RU-Senti dataset, which contain multiple labeled image-text pairs. The classification accuracy (Acc) and F1 score of each model for positive, neutral, and negative sentiment classification on the above three datasets are shown in the table below:

[0087]

[0088] The first column of the table above lists the various models; the second and third columns show the Acc and F1 scores for each model on the MVSA-Single dataset, respectively; the fourth and fifth columns show the Acc and F1 scores for each model on the MVSA-Multiple dataset, respectively; and the sixth and seventh columns show the Acc and F1 scores for each model on the RU-Senti dataset, respectively. For example, the cell in the second row and second column indicates that the VAuLT model has a classification accuracy of 75.60% on the MVSA-Single dataset. This demonstrates that the sentiment detection model 100 exhibits better multimodal sentiment classification performance compared to the two existing models.

[0089] In this embodiment, ablation experiments are further conducted on the RU-Senti dataset to verify the pre-training, cross-modal conversion module, and loss function of the multimodal emotion detection method based on modality synchronization. itc Recorded as ITC loss, Loss itd Recorded as ITD loss, Loss mlm Denoted as MLM loss and Loss itm The impact of ITM loss on the performance of sentiment detection model 100 is shown in the table below:

[0090] Acc F1 score Remove pre-training 81.46％ 0.7855 Remove cross-modal conversion module 82.87％ 0.7989 Remove ITC loss 82.52％ 0.8007 Remove ITD loss 82.26％ 0.7933 Remove MLM loss 82.98％ 0.8022 Remove ITM loss 82.68％ 0.7971 Complete sentiment detection model 84.98％ 0.8401

[0091] The first column in the table above lists the various ablation models, while the second and third columns represent the Acc and F1 scores of each model on the RU-Senti dataset, respectively. For example, the cell in the eighth row and second column indicates that the classification accuracy (Acc) of the complete sentiment detection model 100 obtained using the modality-synchronous multimodal sentiment detection method is 84.98%. This demonstrates that in this embodiment, the pre-training, cross-modal conversion module, and Loss... itc Loss itd Loss mlm and Loss itm All of these can improve the multimodal sentiment classification performance of the model, and the complete sentiment detection model 100 obtained by combining the above features has the best multimodal sentiment classification performance.

[0092] The role and effect of the embodiments

[0093] According to the multimodal sentiment detection method based on modal synchronization involved in this embodiment, on the one hand, a cross-modal transformation module is used to transform a vector from one modality to the semantic space of another modality, allowing the model to compare vectors from different modalities in the same semantic space; on the other hand, a single-modal fusion module is used to synchronize the input in dimensions. In summary, this method can obtain accurate multimodal sentiment classification results.

[0094] Furthermore, a momentum image encoder and a momentum text encoder are set up in the pre-training process. The vectors of the two modalities of the same sample are used as positive samples, and other samples in the training data are used as negative samples. This improves the training sample data without affecting the consistency of sample encoding in unlabeled training, thereby obtaining a model with better performance.

[0095] Furthermore, language masking and image-text matching tasks, along with their corresponding loss functions, are set up during pre-training, enabling the pre-trained model to handle these tasks and further improve model performance.

[0096] Furthermore, the parameters of the multilayer perception module are optimized using labeled training data during fine-tuning, which reduces the need for labeled training data and improves the model's prediction accuracy.

[0097] Those skilled in the art should understand that this invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to this invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal sentiment detection method based on modal synchronization, used to obtain corresponding sentiment detection results based on a specified image and corresponding specified text, characterized in that, Includes the following steps: Step S1: Construct an emotion detection model; Step S2: Construct a training dataset based on existing image and text data, and train the sentiment detection model based on the training dataset to obtain a trained sentiment detection model; Step S3: Input the specified image and the specified text into the sentiment detection model to obtain the sentiment detection result. The emotion detection model includes: An image encoding module is used to encode the specified image to obtain an image vector; An image self-attention module is used to perform self-attention processing on the image vector to obtain an image embedding; The text encoding module is used to encode the specified text to obtain a text vector; A text self-attention module is used to perform self-attention processing on the text vector to obtain the text embedding; An image cross-modal conversion module is used to generate an image modality conversion vector based on the image vector and the text vector, wherein the calculation expression for the image modality conversion vector in the image cross-modal conversion module is: In the formula X v2t Let T be the image modality transformation vector, T be the text vector, and V be the image vector. The weight matrix for the query of the image modality. The weight matrix of the keys for the image modality. Let d be the weight matrix of the image modal values. k Let be the dimension of the key vector. The image fusion module is used to concatenate the image modality transformation vector and the image embedding to obtain the image single-modality embedding; A text cross-modal conversion module is used to generate a text modality conversion vector based on the image vector and the text vector. In this module, the calculation expression for the text modality conversion vector is: In the formula X t2v for, The weight matrix for text-modal queries. This is the weight matrix of the keys in the text modality. The weight matrix represents the values ​​of the text modality; The text fusion module is used to concatenate the text modality transformation vector and the text embedding to obtain a text single modality embedding; The fusion encoding module is used to perform cross-modal attention processing on the image single-modal embedding and the text single-modal embedding to obtain a fusion vector; The multilayer perception module, including a multilayer perceptron, is used to process the fusion vector to obtain the emotion detection result.

2. The multimodal emotion detection method based on modal synchronization according to claim 1, characterized in that: in, In the image self-attention module, the calculation expression for the image embedding is: In the formula X v The image embedding is V, where V is the image vector. The weight matrix for the query of the image modality. The weight matrix of the keys for the image modality. Let d be the weight matrix of the image modal values. k Let be the dimension of the key vector. In the text self-attention module, the calculation expression for the text embedding is: In the formula X t Let T be the text embedding, and T be the text vector. The weight matrix for text-modal queries. This is the weight matrix of the keys in the text modality. This is the weight matrix of the values ​​of the text modality.

3. The multimodal emotion detection method based on modal synchronization according to claim 1, characterized in that: in, In the image fusion module, the calculation expression for the image single-modal embedding is: In the formula For the image single-modal embedding, concat is the concatenation operation, X v For image embedding, X v2t The image modality transformation vector, In the text fusion module, the calculation expression for the text unimodal embedding is: In the formula For the text unimodal embedding, X t For the text embedding, X t2v The text modality transformation vector is denoted as .

4. The multimodal emotion detection method based on modal synchronization according to claim 1, characterized in that: in, In the fusion coding module, the calculation expression for the fusion vector is: In the formula X fusion For the fusion vector, For the text single-modal embedding, For the image single-modal embedding, The weight matrix for text-modal queries. The weight matrix of the keys for the image modality. Let d be the weight matrix of the image modal values. k is the dimension of the key vector.

5. The multimodal emotion detection method based on modal synchronization according to claim 1, characterized in that: in, The multilayer perceptron generates a probability vector based on the fusion vector. The expression for calculating the probability vector is: P=softmax(WX fusion +b), In the formula X fusion Let be the fusion vector, W be the weight matrix, and b be the bias vector.

6. The multimodal emotion detection method based on modal synchronization according to claim 1, Its features are: in, Step S2 includes the following sub-steps: Step S2-1: Construct a training dataset based on existing image and text data; Step S2-2: Pre-train the sentiment detection model based on the training dataset to obtain a pre-trained sentiment detection model; Steps S2-3: Fine-tune the pre-trained sentiment detection model based on the training dataset to obtain the trained sentiment detection model. The training dataset includes multiple image-text pairs consisting of training images, corresponding training text, and corresponding ground truth labels. In step S2-2, the overall loss is calculated based on the training dataset, and the parameters of the image encoding module, the image self-attention module, the text encoding module, the text self-attention module, the image cross-modal conversion module, and the text cross-modal conversion module in the sentiment detection model are updated based on the overall loss. In steps S2-3, the loss is calculated based on the training dataset, and the parameters of the multilayer perceptron in the pre-trained sentiment detection model are updated based on the loss.

7. The multimodal emotion detection method based on modal synchronization according to claim 6, characterized in that: in, The image encoding module includes an image encoder. The text encoding module includes a text encoder. The formula for calculating the total loss is as follows: Loss=Loss itc +Loss itd +Loss mlm +Loss itm , Loss itm =-(y itm logp itm +(1-y itm log(1-p itm )), In the formula, Loss is the total loss, |queue t | represents the length of the queue of text vectors obtained by the momentum text encoder, |queue v | represents the length of the queue of image vectors obtained by the momentum image encoder. The image modality transformation vector corresponding to the training image in the i-th image-text pair. The text vector corresponding to the training text in the i-th image-text pair. This is the vector generated after processing the j-th training text in the text vector queue using a momentum text encoder. The text modality transformation vector corresponding to the training text in the i-th image-text pair. The image vector corresponding to the training image in the i-th image-text pair. P(X) is the vector generated by processing the j-th training image in the image vector queue using a momentum image encoder, where τ is the temperature parameter. v Q(X) is the output obtained by the image encoder processing the training images in the image-text pair. v P(X) is the output obtained by the momentum image encoder after processing the training image. t Q(X) is the output obtained by the text encoder processing the training text in the image-text pair. t ) represents the output obtained by the momentum text encoder processing the training text, and V represents the size of the vocabulary. For the one-hot encoding of the i-th word, The probability that the word at the mask position in the sentiment detection model is the i-th word in the vocabulary is predicted. itm p is the ground truth label corresponding to the image-text pair input to the sentiment detection model. itm The predicted probability output generated by the sentiment detection model based on the image-text pair is the following. The momentum image encoder has the same structure as the image encoder. The momentum image encoder updates momentum based on the parameters of the image encoder. The momentum text encoder has the same structure as the text encoder. The momentum text encoder updates momentum based on the parameters of the text encoder.

8. The multimodal emotion detection method based on modal synchronization according to claim 6, characterized in that: in, The expression for calculating the loss is: In the formula, Loss is the loss, N is the total number of image-text pairs, M is the total number of sentiment categories corresponding to the true labels, and P... i,j The probability that the i-th image-text pair of the input belongs to the j-th sentiment category, as predicted by the pre-trained sentiment detection model.

Images