A multi-modal meme graph sentiment detection method

Abstract

The application discloses a kind of multi-modal meme graph sentiment detection methods, the present application has carried out fine data preprocessing work to meme graph data, and with VL-BERT, UNITER, Villa these 3 kinds of multi-modal pre-training model as foundation, through input entity, character feature information, introduce the loss function and classification head of improvement hierarchical multi-label classification by introducing adversarial training, multi-modal pre-training model has been improved.The improved model has good effect on meme graph sentiment detection task and sentiment type fine classification task, and the performance exceeds all multi-modal basic classification models and its improved version in the experiment.

Description

Technical Field

[0001] This invention belongs to the field of semantic recognition, and specifically relates to a multimodal meme graph sentiment detection method. Background Technology

[0002] With the popularity of memes on social media platforms, more and more people are using memes to express different emotions online. This has made it necessary to develop multimodal meme detection models that can automatically identify the emotions expressed in memes and classify them into finer emotional types. Meme sentiment detection is a binary classification problem where, given a meme, the model can determine whether the meme expresses a particular emotion. Fine-grained classification of meme emotion types is a hierarchical multi-label classification problem. Given a meme, the model not only needs to determine whether the meme expresses a particular emotion, but also needs to further determine the fine-grained emotion type of the meme. For example, fine-grained emotion types for positive emotions include agreement, admiration, excitement, and gratitude; fine-grained emotion types for negative emotions include anger, sadness, and disappointment; and fine-grained emotion types for neutral emotions include confusion, curiosity, and surprise.

[0003] To perform meme graph sentiment detection, a common approach is to fine-tune a multimodal pre-trained model on a meme graph dataset. For this binary classification problem, the multimodal pre-trained model uses a multilayer perceptron with an output dimension of 1 as its classification head during fine-tuning, and employs binary cross-entropy loss as the loss function. For hierarchical multi-label classification problems involving fine-grained sentiment type classification, it is generally decomposed into multiple binary classification problems. During fine-tuning, the multimodal pre-trained model uses a multilayer perceptron with an output dimension of c, where c is the number of sentiment types the model can recognize, and the loss function is the sum of the c binary cross-entropy losses. Furthermore, to ensure the model output satisfies hierarchical constraints, necessary post-processing steps are required. A common post-processing method is maximum constraint, which ensures that the probability value of the parent class output is the maximum of the probabilities of all its child classes, and that each class is also a child class of itself. To further improve the sentiment detection performance of the model in meme graphs, Zhu R et al. performed meticulous data preprocessing on meme graph data in their work "Enhance multimodal transformer with external label and in-domain pretrain: Hateful meme challenge winning solution" (arXiv preprint arXiv:2012.08290). They extracted the feature information of entities and people appearing in the meme graphs and input it into the multimodal pretraining model. At the same time, for the VL-BERT multimodal pretraining model, they proposed an extended VL-BERT structure to promote the model's cross-modal fusion capability. These improvements enhanced the meme graph detection performance of the multimodal pretraining model.

[0004] Furthermore, to maximize the performance of meme graph sentiment detection models, a common approach is to integrate models based on different types of multimodal pre-trained models. For example, in the work of Zhu R et al., model integration was performed based on base and large versions of four multimodal pre-trained models: VL-BERT, UNITER, Villa, and ERNIE-ViL.

[0005] For hierarchical multi-label classification tasks such as fine-grained sentiment classification, using a multilayer perceptron as the classification head fails to allow the model to extract the necessary hierarchical relationships between categories, and using binary cross-entropy loss cannot handle the imbalanced distribution of positive and negative samples common in hierarchical multi-label classification tasks. Furthermore, the superior performance of ensemble models depends on a large number of diverse base learners; ensemble models based on different types of multimodal pre-trained models can only train a limited number of base models, thus limiting the performance of ensemble models. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, the purpose of this invention is to construct a multimodal meme graph sentiment detection model that can perform the following two classification tasks:

[0007] (1) Meme graph sentiment detection, that is, given a meme graph, the model can complete the binary classification problem of whether the meme graph expresses a certain sentiment, such as a benevolent attitude or a malicious attitude, especially the trend of benevolent or malicious attitudes formed by a certain group.

[0008] (2) Meme graph sentiment type fine-grained classification, i.e., the hierarchical multi-label classification problem of meme graphs. Given a meme graph, the model not only needs to determine whether the meme graph expresses the stated sentiment, but also needs to further determine whether the meme graph contains fine-grained sentiment of the stated sentiment. The category hierarchy of the second classification task is as follows: Figure 1 As shown, emotion 1 is the emotion type, and emotions 2, 3, 4, and 5 are fine-grained emotion types. Emotion 1 is the parent category of emotions 2, 3, 4, and 5. It is important to note that when the model performs inference, the prediction results must satisfy the hierarchical relationship between categories. That is, if the meme graph identification result of the emotion is negative, then the prediction result of the fine-grained emotion type must also be negative.

[0009] This invention is based on four multimodal pre-trained models: VL-BERT, UNITER, Villa, and ERNIE-ViL. It improves these models from multiple perspectives and finally constructs a meme graph sentiment detection model through model ensemble. The improvements include incorporating entity and person feature information from the meme graph, adding FreeLB adversarial training during the fine-tuning phase of the multimodal model, introducing Focal loss, asymmetric loss, negative sample tolerance regularization, and maximum value constraint loss to improve the loss function, and introducing HMCN to improve the classification head. Furthermore, for the hierarchical multi-label classification task of fine-grained sentiment classification, this invention proposes two new classification head structures: MR-head and P-head. These structures respectively introduce hidden vector mean regularization and parent class probability priors to allow the model to more fully utilize the hierarchical relationships between categories. Focal loss, asymmetric loss, negative sample tolerance regularization, and maximum value constraint loss are used to address the imbalance of positive and negative sample distributions, while HMCN, MR-head, and P-head help the model extract the necessary hierarchical relationships between categories. Through the above-mentioned improvement strategies, this invention constructs 31 different base models for the meme graph sentiment detection task and 111 different base models for the fine-grained sentiment type task. Through model ensemble, the performance of the meme graph sentiment detection model is superior to that of all base models.

[0010] The solution of the present invention:

[0011] Figure 2 This is the main flowchart of the technical solution of this invention. For example... Figure 2 As shown, a multimodal meme graph sentiment detection method includes the following steps:

[0012] Step 1: Remove the text content from the meme image to obtain meme image A;

[0013] Step 2: Use the trained Faster R-CNN model to extract the features of the region of interest (ROI) of the meme image A from Step 1. When the meme image A contains multiple sub-images, use the trained Faster R-CNN model to first separate the multiple sub-images in the meme image A, and then regard the region where each sub-image is located as the ROI. Extract the features of the corresponding ROI through the RoI Pooling module of Faster R-CNN.

[0014] Step 3: Identify entity information in meme image A. If meme image A contains multiple sub-images, identify entity information in the sub-images. Use a face detector to identify the feature information of people in meme image A. If meme image A contains multiple sub-images, identify the feature information of people in the sub-images.

[0015] Step 4: Train a basic multimodal classification model based on VL-BERT, UNITER, and Villa, with a multilayer perceptron as the classification head; train 6 models each for the meme graph sentiment detection task and the sentiment type fine classification task by training the base and large versions of VL-BERT, UNITER, and Villa.

[0016] Step 5: Improve the basic multimodal classification model from Step 4. The specific improvement plan is as follows:

[0017] Option 1 involves inputting the entity information and character feature information from step 3 into the model in step 4 as text; 12 models were trained for each of the meme graph sentiment detection task and the sentiment type fine classification task.

[0018] Option 2 involves adding adversarial training to the language modality of the base version of VL-BERT in step 4, and adding adversarial training to the language, vision, and language and vision modalities of the base versions of UNITER and Villa, training 7 models each for the meme graph sentiment detection task and the sentiment type fine classification task.

[0019] Option 3 uses FL, ASL, NTR-FL and MC loss functions respectively when training the base and large versions of VL-BERT, UNITER and Villa in step 4, and trains 24 models for the sentiment type fine classification task.

[0020] Option 4 uses MR-head and P-head classification heads respectively when training the base and large versions of VL-BERT, UNITER and Villa in step 4, training 12 models for the fine classification task of sentiment type.

[0021] The features of the region of interest from step 2 are input into the multimodal classification model obtained by the improved scheme to obtain the detection results for the corresponding task.

[0022] Furthermore, step 5 also includes scheme five, which uses the basic multimodal classification model from step 4 and the improved multimodal classification models from schemes one to four in step 5 as base models. Multiple base models are selected, and the average of the predicted logit values ​​of each base model is used to integrate them. The integrated model is the multimodal meme graph sentiment detection model; the integration method is shown in the following formula.

[0023]

[0024] in, The ensemble model predicts the logit value for sample i. Let K be the predicted logit value of the k-th base model for sample i, where K is the number of base models selected.

[0025] Further, in step 1, the removal of text content from the meme image specifically involves: firstly, using OCR to detect the text location in the meme image, then covering the text in the meme image and using DeepFillv2 image inpainting technology to remove the text content from the image.

[0026] Further, in step 2, the specific training method of the trained Faster RCNN model is as follows: multiple images are randomly selected from the GQA dataset to obtain the training dataset, and the Faster RCNN model is trained; before the meme image A is input into the trained Faster RCNN model, data augmentation is performed, and the data augmentation method includes one or both of random horizontal flipping and random image size scaling.

[0027] Furthermore, in step 3, the network entity detection API is used to identify entity information in meme image A; the FairFace face detector is used to identify the race and gender of people in the image. This face detector can identify 8 races and 2 genders. The network entity detection API is such as the network entity detection API provided by Google Cloud.

[0028] Further, in step 4, the classification head specifically consists of: taking the multimodal meme graph feature vector f as input, passing it sequentially through a fully connected layer (FC), a Gaussian error linear unit (GELU) activation function, a layer normalization layer (LN), a dropout layer, and another fully connected layer (FC), finally outputting the logit value of the meme graph belonging to each category. Where C is the number of categories in the classification task; the binary cross-entropy loss function is used during training.

[0029] Furthermore, in step 5, the first scheme is as follows: after adding entity information and person characteristic information, the text input is " <cls>meme text <sep>entity tags <sep>character feature tags <sep>",in, <cls>and <sep>For special input terms, "meme text" refers to the meme graph text, and "entity tags" refers to the text composed of the individual entity names in the meme graph. Different entity names are separated by... <sep>Separated by "character feature tags," these are texts composed of characteristic information about individuals in the meme image. The characteristic information of different individuals is separated by [missing information]. <sep>Separate.

[0030] Furthermore, the second option for step 5 is as follows:

[0031] Loss function with perturbation added to visual modality and the loss function that adds perturbations to the language modality As shown in the following formula:

[0032]

[0033]

[0034] If perturbations are added to both the visual and language modalities simultaneously, the loss function during adversarial training will change. As shown in the following formula:

[0035]

[0036] Where, x img and x txt The inputs are visual and linguistic modalities, respectively, δ img and δ txt These represent perturbations in the visual and linguistic modalities during adversarial training, respectively, where y is the sample label and L is the perturbation. std To avoid using the standard classification loss function during adversarial training, and to ensure the model can still output the correct label even with added input perturbations, L kl (p,q)=KL(p||q)+KL(q||p), where KL(·) is the KL divergence, used to ensure that the logit value of the output remains consistent before and after adding input perturbations. This is the logit value of the model output without any input perturbation.

[0037] Furthermore, the third option in step 5 specifically refers to:

[0038] The Focal loss function (FL) is shown in the following equation:

[0039]

[0040] L(p,y)=-yL + (p)-(1-y)L - (p)

[0041]

[0042] Where N is the dataset size and C is the number of categories; Let i be the logit value of category k corresponding to sample i. Let σ(z) be the label of category k corresponding to sample i, and σ(z) = 1 / (1+e -z ) is the sigmoid function, γ is the focus factor, which controls the degree of decay of the weights of simple samples, and α is a hyperparameter, which is used to balance the weights of positive and negative samples;

[0043] The ASL (Asymmetric Loss) loss function is shown in the following equation:

[0044]

[0045] L(p,y)=-yL + (p)-(1-y)L - (p)

[0046]

[0047] p m =max(pm,0)

[0048] Where, γ + and γ - These are the focus factors for positive and negative samples, respectively, used to control the degree of decay of the weights of simple samples in positive and negative samples, and m is the probability offset threshold.

[0049] The NTR-FL (Negative Sample Tolerance Regularization based on Focal loss) loss function is shown in the following equation:

[0050]

[0051]

[0052]

[0053] L(p,y)=-yL + (p)-(1-y)L - (p)

[0054]

[0055] Where λ is the negative sample tolerance regularization strength; p is the intrinsic bias of the model. k =n k / N represents the class probability prior. The number of positive samples in category k; These are hyperparameters used to embed the model's intrinsic biases into the model training process; k These are the normalization coefficients;

[0056] The MC (Maximum Limiting Loss) loss function is the modified loss function, which calculates the probability value of a sample belonging to the positive class. Its specific form is shown in the following formula:

[0057]

[0058] in, Let be the probability that sample i belongs to category j. Let D be the probability that sample i belongs to class k after modification by the maximum limit loss. k Let D be the set of subclasses of class k, where k∈D k For BCE,

[0059] Furthermore, the fourth option in step 5 is specifically as follows:

[0060] The MR-head classification head is represented as a tree structure G. G adds a new leaf node to each non-leaf node of the category hierarchy tree T (excluding the root node). This leaf node is a special child node of each non-leaf node. The root node in G corresponds to the input feature vector f of the MR-head classification head. Each other non-leaf node corresponds to a hidden activation output vector a in the MR-head. Each leaf node corresponds to the output logit value z of a certain category. Each edge corresponds to a network module, whose input is the feature vector represented by the node at the tail of the edge, and whose output is... The edge head corresponds to the feature vector represented by the node; each edge pointing to a non-leaf node corresponds to a basic fully connected block, which is composed of a cascaded FC layer, a GELU layer, an LN layer, and a Dropout layer; each edge pointing to a leaf node corresponds to a fully connected layer with an output dimension of 1; the logit value of the category corresponding to the leaf node in T is the output of the leaf node at the same position in G, and the logit value of the category corresponding to the non-leaf node in T is the output of the special child node of the non-leaf node at the same position in G; the logit value of category C (i.e., the leaf node in T) is z. C The logit value of category A (i.e., non-leaf nodes in T) is z. A In hierarchical multi-label classification tasks, the logit value of the subclass corresponding to a sample is always less than or equal to the logit value of the parent class. This means that the logit value of the subclass output by the model is influenced to some extent by the logit value of the parent class. Intuitively, the logit values ​​of the subclass and parent classes should be positively correlated. To incorporate this prior knowledge into the model, the MR-head also applies regularization to the hidden vector h in the network. Specifically, the output of the FC layer in the basic fully connected block is the hidden vector h of the non-leaf nodes. The MR-head calculates the mean of each element in h and then adds the resulting mean to the output z corresponding to the special child node of the non-leaf node. Figure 6 In the diagram, the dashed arrows represent the process of adding the mean of the hidden vector of a non-leaf node to its special child node. This process can be expressed by the following formula:

[0061]

[0062] Where z is the original logit value, i.e., the output corresponding to the special child node of the non-leaf node, h is the hidden vector of the non-leaf node, and m is the output dimension of the hidden layer. This represents the final logit value; let a be the non-leaf node corresponding to category c. c a c The special child node is z c a c The hidden vector is h c During training, optimizing the loss function will make h c The mean of h is tied to the true label value of category c. When a sample belongs to c, h c The mean will be too large when the sample does not belong to c, h c The mean will be smaller, which is equivalent to implicitly affecting h. c Regularization was applied, making h c Related to the logit value of c. h after regularization. c The forward propagation of the network further influences the logit values ​​of subsequent categories, i.e., the logit values ​​of all subcategories of c. Therefore, applying regularization to the hidden vectors ensures that the logit value of all subcategories in the MR-head is influenced by the logit value of their parent category, allowing the model to more fully utilize the hierarchical relationships between categories. For the sentiment type fine-grained classification task, the network structure of the MR-head is as follows: Figure 7 As shown, A represents a certain emotion category, and B, C, D, and E are the fine-grained emotion subcategories corresponding to the emotion category.

[0063] The P-head classification head, based on the MR-head, further introduces a prior probability of the parent class through the total probability formula to strengthen the connection between the output probabilities of the sub-class and the parent class, thereby enhancing the interpretability of the model and improving its hierarchical multi-label classification performance. Let the parent class be A and the sub-class be B. Using the total probability formula, the probability that sample i belongs to class B can be expressed as follows:

[0064]

[0065] in, This represents the subclass logit value output by the MR-head, representing the label of sample i in class j. Logit value of parent category The impact will As Treat as prior probability of parent category Therefore, in the P-head, the probability that sample i belongs to class B is expressed as follows:

[0066] However, directly Using the parent category prior probability presents the following problem: when sample i belongs to parent category A but not child category B, optimizing the loss function for child category B becomes problematic. This will cause the model to update its parameters. This decrease in probability can impair the model's prediction performance for parent category A. To address this issue, P-head modifies the formula for calculating the prior probability of the parent category, as shown in the following equation.

[0067]

[0068] Where α≥0 is a hyperparameter used to evaluate the logit value of the parent category. Perform a linear transformation. The graph of the parent class prior probability relative to the parent class logit value for different values ​​of α is shown below. Figure 8 As shown, the larger α is, the more parent class logit values ​​fall within the non-linear region of the sigmoid function in the interval (-α, +∞). For these parent class logit values ​​falling within the non-linear region, the gradient of the sub-class loss function with respect to them is very small. Therefore, optimizing the sub-class loss function will not have a significant impact on their values. In other words, when a sample belongs to the parent class but not the sub-class, the larger parent class logit value output by the model will not decrease due to optimizing the sub-class loss function, and thus will not affect the model's detection performance for the parent class. Furthermore, since the logit value output by the model is generally near 0, when α is large enough, the performance of P-head is similar to that without parent class probability prior.

[0069] Compared with the prior art, the beneficial effects of the present invention are:

[0070] This invention constructs a multimodal meme graph sentiment detection method, capable of meme graph recognition of sentiments and fine-grained classification of sentiment types. To build the model, this invention performs meticulous data preprocessing on the meme graph data and, based on three multimodal pre-trained models—VL-BERT, UNITER, and Villa—introduces adversarial training by inputting entity and person feature information, improving the loss function and classifier head of hierarchical multi-label classification, thus enhancing the multimodal pre-trained model. Simultaneously, this invention proposes two novel hierarchical multi-label classification heads, MR-head and P-head, whose performance is leading among all improved methods. Finally, this invention constructs a meme graph sentiment detection model using a model ensemble method. Experiments show that, thanks to the diverse base learners constructed through various improved methods, this model achieves good results in both meme graph sentiment recognition and fine-grained sentiment type classification tasks, outperforming all multimodal basic classification models and their improved versions in the experiment. Attached Figure Description

[0071] Figure 1 The category hierarchy for the second classification task;

[0072] Figure 2 This is a flowchart of the technical solution of the present invention;

[0073] Figure 3 Basic classification head structure;

[0074] Figure 4 A schematic diagram of the extended VL-BERT structure;

[0075] Figure 5 This is a schematic diagram of the HMCN-F structure used for the second classification task.

[0076] Figure 6 This is a tree-like representation of the category hierarchy and the corresponding MR-head.

[0077] Figure 7 This is a schematic diagram of the MR-head structure used for the second classification task.

[0078] Figure 8 This is a graph showing the prior probability of the parent class as a function of the parent class logit value when the values ​​are different. Detailed Implementation

[0079] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0080] All experiments below were conducted on a TITAN XP with 12GB of video memory. In the experiments, this invention trained corresponding base classification models and improved classification models based on different multimodal pre-trained models, including base and large versions of VL-BERT, UNITER, and Villa. To more accurately measure the generalization performance of the models, each model was trained for at least 8 epochs on the training set, with the model saved once per epoch. The epoch with the best performance on the validation set was recorded, and the performance of the model saved in that epoch on the test set was taken as the final performance of the model on the test set. To ensure reproducibility, a fixed random number seed was used during the experiments.

[0081] The meme graph sentiment detection model proposed in this invention uses the meme graph dataset provided by the SemEval-2022 semantic evaluation competition. This dataset consists of 10,000 meme graphs, of which 8,500 are used as the training set, 500 as the validation set, and 1,000 as the test set. The dataset contains five different sentiment categories, with the latter four being fine-grained sentiment categories of the first sentiment category. The number of positive and negative samples in each category is shown in the table below.

[0082] Table 1 shows the distribution of positive and negative examples for each category in the meme graph dataset.

[0083]

[0084] Due to the imbalanced distribution of positive and negative samples, this invention does not use classification accuracy as the model performance evaluation metric, but instead uses AUROC. Compared to accuracy, AUROC can measure the quality of the model's sample score ranking, and can more objectively reflect the model's classification performance in the case of class imbalance. For the multi-label classification in Task 2, the model evaluation metric is the average AUROC of all 5 classes.

[0085] Example 1: Adding extra input information and adversarial training

[0086] Table 2 shows the performance of the improved classification model on the meme graph test set, which incorporates additional entity and person feature information (race and gender) as input and includes adversarial training. Each item in the table represents the AUROC of the classification model based on a certain pre-trained model on the test set and its relative performance improvement compared to the base model. Since adversarial training has high requirements for GPU memory and computation, this embodiment does not include adversarial training for the large version model. Furthermore, because Fast R-CNN in VL-BERT is trained online, it places very high demands on the model's GPU memory. Therefore, this invention only adds adversarial training for the language modality of the base version model based on VL-BERT. During adversarial training, the number of training steps K = 3, and the training step size α = 1.0 × 10⁻⁶. -3 The perturbation boundary ∈ = +∞, and the initial value of the perturbation is directly set to 0.

[0087] Table 2 shows the performance of the model with additional input information and adversarial training on the meme graph test set.

[0088]

[0089]

[0090] Note: 1) Each item in the table represents the AUROC of a classification model based on a certain pre-trained model on the test set and its relative performance improvement relative to the base model, where the relative performance improvement is the percentage increase in the AUROC of the classification model compared to the base classification model. 2) "vanilla" is the base classification model, "entity" is the improved model with entity information added, "character feature" is the improved model with character feature information added, "adv-text" is the improved model with adversarial training on the language modality, "adv-img" is the improved model with adversarial training on the visual modality, and "adv-text-img" is the improved model with adversarial training on both the language and visual modalities. 3) The base / large subscript indicates the size of the multimodal pre-trained model, and the multilabel subscript indicates that the model is a hierarchical multi-label classification model used for Task 2. 4) The bolded items in the table represent the AUROC and relative performance improvement of the best-performing model among the base and improved models based on different multimodal pre-trained models.

[0091] The average relative performance improvement of the models with added input information and adversarial training compared to the original multimodal pre-trained models for VL-BERT, UNITER, and Villa is shown in the table below.

[0092] Table 3 shows the average relative performance improvement of the model with added additional input information and adversarial training on the meme graph test set.

[0093]

[0094] Note 1) The bolded items in the table represent the average relative performance improvement corresponding to the improved model that improves the average relative performance.

[0095] As shown in Table 3, introducing adversarial training into multimodal pre-trained models can improve multimodal model performance at the cost of increased training overhead. For models with added input information, they did not improve average relative performance. However, for VL-BERT, due to its extended VL-BERT structure, it utilizes visual feature embedding vectors to perform cross-modal alignment of language and visual modal information when inputting entity and person feature information. Therefore, adding additional input information can improve the performance of VL-BERT-based models, and the improvement is more significant for the large version model. Adding person feature information improved the performance of the large version of VL-BERT by 0.14% and 0.65%, respectively.

[0096] Example 2: Improved Loss Function

[0097] The performance of the improved loss function model on the meme graph test set is shown in Table 4. In the experiments, the hyperparameters in the various improved loss functions were taken from the empirical values ​​in the original paper. Specifically, α = 0.25 and γ = 2 were used in Focal loss, and γ was taken as γ in ASL. + =0, γ - =4, m=0.05, λ=2 in NTR.

[0098] Table 4 shows the performance of the model with improved loss function on the meme graph test set.

[0099]

[0100] Note 1) "vanilla" is the basic classification model, "FL" is the improved model using Focal loss as the loss function, "ASL" is the improved model using asymmetric loss as the loss function, "NTR-FL" is the improved model using Focal loss with negative sample tolerance regularization as the loss function, and "MC" is the improved model using maximum limit loss as the loss function.

[0101] Table 5 shows the average relative performance improvement of the models with improved loss functions compared to the original multimodal pre-trained models for VL-BERT, UNITER, and Villa.

[0102] Table 5 shows the average relative performance improvement of the model with the improved loss function on the meme graph test set.

[0103]

[0104] As shown in Table 5, improving the loss function for the hierarchical multi-label classification problem in Task 2 can effectively improve the classification performance of the model. Among them, the model based on NTR-FL has the largest performance improvement, reaching 0.48%. These four loss functions improve the model performance from different perspectives: FL attenuates the weight of simple samples, ASL attenuates the weight of simple negative samples, NTR alleviates the over-suppression of negative samples, and MC introduces a hierarchical structure of categories into the model.

[0105] Example 3: Improved Sorting Head

[0106] The performance of the improved classification head model on the meme graph test set is shown in Table 6. Specifically, this invention performed hyperparameter search on six base and large versions of the P-head model (VL-BERT, UNITER, and Villa). For each model, three different values ​​of α were searched: α=3, α=3.5, and α=4. Each value was trained for eight epochs. The performance of the model with the best performance on the validation set across all 24 epochs was then used as the final performance of the improved P-head model on the test set. For the three different classification heads used in the experiment—Multilayer Perceptron, MR-head, and P-head—this invention artificially ensured that the number of structural parameters was approximately the same for all three structures when selecting the output dimension of the hidden layer, thus achieving a fair comparison.

[0107] Table 6 shows the performance of the improved classification head model on the meme graph test set.

[0108]

[0109]

[0110] Note 1) "vanilla" is the basic classification model, "MR-head" is the classification head using an improved model of MR-head, and "P-head" is the classification head using an improved model of P-head.

[0111] Table 7 shows the average relative performance improvement of the improved classification head models compared to the original multimodal pre-trained models for VL-BERT, UNITER, and Villa. The MR-head, which extracts fine-grained node and edge-level class hierarchical relationships, brings an average relative performance improvement of 0.99%. The P-head, which incorporates parent class prior probabilities, further enhances the performance, bringing an average relative performance improvement of 1.36%. It is worth noting that the performance improvement brought by the proposed MR-head is second only to adversarial training on the visual modality, but the computational cost of MR-head is far less than that of adversarial training, making it a more cost-effective improvement method. Furthermore, the performance improvement brought by the proposed P-head is the largest among all improvement methods, making it the most effective improvement method.

[0112] Table 7 shows the average relative performance improvement of the model with improved classification heads on the meme graph test set.

[0113]

[0114] Example 4: Model Integration

[0115] When ensemble models, a large number of different base models can greatly improve the final performance of the ensemble model. Therefore, this invention does not deliberately select the best-performing models for model ensemble. Instead, it uses the basic classification models based on each multimodal pre-trained model and all improved classification models as base models and integrates them together to build the final multimodal meme graph sentiment detection model.

[0116] For the memes of Task 1 Figure 2 For classification tasks, this invention integrates 25 base models. As shown in Table 8, the base models include 6 basic classification models based on all 6 pre-trained models and 12 improved models with added information, 4 improved models based on UNITER base and Villa base that perform adversarial training on both visual and visual-language modalities, and 3 improved models based on VL-BERT base, UNITER base and Villa base that perform adversarial training on the language modality.

[0117] For the meme graph hierarchical multi-label classification task in Task 2, this invention integrates 61 base models, as shown in Table 9. The base models include 6 basic classification models based on all 6 pre-trained models, 12 improved models with added information, 24 improved models with improved loss functions, and 12 improved models with improved classification heads. There are also 4 improved models based on the UNITER base and Villa base that perform adversarial training on the visual and visual-language modalities, and 3 improved models based on the VL-BERT base, UNITER base, and Villa base that perform adversarial training on the language modality.

[0118] Table 8 lists the 25 base models used for Task 1.

[0119]

[0120] Note 1) √ in the table indicates that the ensemble model includes a base or improved classification model based on the pre-trained model.

[0121] Table 9 lists the 61 base models used for Task 2.

[0122]

[0123] The performance of the ensemble model on the meme graph test set is shown in Table 10. The ensemble model outperformed all base models in both the binary classification of whether the meme graph contains the sentiment in Task 1 and the fine classification of the sentiment type in Task 2. Such excellent performance is also due to its large number of different base models. It is by combining the strengths of each model that the performance of the ensemble model is further improved compared to a single base model.

[0124] Table 10 Performance of the ensemble model on the meme graph test set

[0125]

[0126] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other.

[0127] This document uses specific examples to illustrate the principles and implementation process of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.< / sep> < / sep> < / sep> < / cls> < / sep> < / sep> < / sep> < / cls>

Claims

1. A multimodal meme graph sentiment detection method, characterized in that, The steps include the following: Step 1: Remove the text content from the meme image to obtain meme image A; Step 2: Use the trained Faster R-CNN model to extract the features of the region of interest (ROI) of the meme image A from Step 1. When the meme image A contains multiple sub-images, use the trained Faster R-CNN model to first separate the multiple sub-images in the meme image A, and then regard the region where each sub-image is located as the ROI. Extract the features of the corresponding ROI through the RoI Pooling module of Faster R-CNN. Step 3: Identify entity information in meme image A. If meme image A contains multiple sub-images, identify entity information in the sub-images. Use a face detector to identify human features in meme image A. If meme image A contains multiple sub-images, identify human features in the sub-images. Step 4: Train a basic multimodal classification model based on VL-BERT, UNITER, and Villa, with a multilayer perceptron as the classification head; train 6 models each for the meme graph sentiment detection task and the sentiment type fine classification task by training the base and large versions of VL-BERT, UNITER, and Villa. Step 5: Improve the basic multimodal classification model from Step 4. The specific improvement plan is as follows: Option 1 involves inputting the entity information and character feature information from step 3 into the model in step 4 as text; training 12 models each for the meme graph sentiment detection task and the sentiment type fine classification task. Option 2 involves adding adversarial training to the language modality of the base version of VL-BERT in step 4, and adding adversarial training to the language, vision, and language and vision modalities of the base versions of UNITER and Villa, training 7 models each for the meme graph sentiment detection task and the sentiment type fine classification task. Option 3: When training the base and large versions of VL-BERT, UNITER and Villa in step 4, use FL, ASL, NTR-FL and MC loss functions respectively to train 24 models for the sentiment type fine classification task. Option 4: When training the base and large versions of VL-BERT, UNITER and Villa in step 4, use MR-head and P-head classification heads respectively to train 12 models for the fine classification task of sentiment type. The features of the region of interest from step 2 are input into the multimodal classification model obtained by the improved scheme to obtain the detection results for the corresponding task.

2. The multimodal meme graph detection method according to claim 1, characterized in that, Step 5 also includes Scheme 5, which uses the basic multimodal classification model from Step 4 and the improved multimodal classification models from Schemes 1 to 4 in Step 5 as base models. Multiple base models are selected, and the average of the predicted logit values ​​of each base model is used to integrate them. The integrated model is the multimodal meme graph sentiment detection model; the integration method is shown in the following formula. in, For the ensemble model to sample The predicted logit value, For the first Each base model for samples The predicted logit value, The number of base models to select.

3. The multimodal meme graph sentiment detection method according to claim 1, characterized in that, Step 1, the removal of text content from the meme image, specifically involves: first, using OCR to detect the text location in the meme image, then covering the text in the meme image and using DeepFillv2 image inpainting technology to remove the text content from the image.

4. The multimodal meme graph sentiment detection method according to claim 1, characterized in that, Step 2, the specific training method of the trained Faster R-CNN model is as follows: multiple images are randomly selected from the GQA dataset to obtain the training dataset, and the Faster R-CNN model is trained; before the meme image A is input into the trained Faster R-CNN model, data augmentation is performed, and the data augmentation method includes one or both of random horizontal flipping and random image size scaling.

5. The multimodal meme graph sentiment detection method according to claim 1, characterized in that, In step 3, the entity information in meme image A is identified using the Network Entity Detection API; and the human features in the image are identified using the FairFace face detector.

6. The multimodal meme graph sentiment detection method according to claim 1, characterized in that, Step 4, the classification head specifically refers to: using the feature vector of the multimodal meme graph. The input is passed sequentially through a fully connected layer (FC), a Gaussian error linear unit (GELU) activation function, a layer normalization layer (LN), a dropout layer, and another fully connected layer (FC). The final output is the logit value of the memogram belonging to each category. ,in, The number of categories for the classification task; a binary cross-entropy loss function is used during training.

7. The multimodal meme graph sentiment detection method according to claim 1, characterized in that, The specific implementation of step 5, Scheme 1, involves adding entity information and person characteristic information, with the text input being " <cls>meme text <sep>entity tags <sep>character feature tags <sep>",in, <cls>and <sep>For special input terms, "meme text" refers to the meme graph text, and "entity tags" refers to the text composed of the individual entity names in the meme graph. Different entity names are separated by... <sep>Separated by "character feature tags," these are texts composed of characteristic information about individuals in the meme map. The characteristic information of different individuals is separated by […]. <sep> Separate.< / sep> < / sep> < / sep> < / cls> < / sep> < / sep> < / sep> < / cls> 8. The multimodal meme graph sentiment detection method according to claim 1, characterized in that, The second option in step 5 is as follows: Loss function with perturbation added to visual modality and the loss function that adds perturbations to the language modality As shown in the following formula: If perturbations are added to both the visual and language modalities simultaneously, the loss function during adversarial training will change. As shown in the following formula: in, and They are visual and linguistic modal inputs, respectively. and These are the visual and linguistic modal perturbations in adversarial training, respectively. For sample labels, This is a standard classification loss function used in adversarial training, designed to ensure the model can still output the correct label even when input perturbations are added. , This is the KL divergence, used to ensure that the logit value of the model's output remains consistent before and after adding input perturbations. This is the logit value of the model output without any input perturbation.

9. The multimodal meme graph sentiment detection method according to claim 1, characterized in that, The specific solution three in step 5 is as follows: The FL loss function is shown in the following formula: in, For the dataset size, Number of categories; For the sample Corresponding category The logit value, For the sample Corresponding category The tag, For the sigmoid function, This is a focus factor used to control the degree of decay of simple sample weights. These are hyperparameters used to balance the weights of positive and negative samples; The ASL loss function is shown in the following formula: in, and These are focus factors for positive and negative samples, respectively, used to control the degree of weight decay for simple samples in positive and negative samples. This is the probability offset threshold; The NTR-FL loss function is shown in the following formula: in, Tolerance for negative samples and regularization strength; This is the model's inherent bias. For class probability priors, For category The number of positive samples; These are hyperparameters used to embed the model's intrinsic biases into the model training process; These are the normalization coefficients; The MC loss function, which is the probability value of a sample belonging to the positive class calculated in the modified loss function, has the following specific form: in, For the sample Category The probability, The sample after being modified with maximum limit loss Category The probability, For category A collection of subclasses, and For BCE, .

10. The multimodal meme graph sentiment detection method according to claim 1, characterized in that, The fourth option in step 5 is as follows: The MR-head classification head is represented as a tree structure G. G adds a new leaf node to each non-leaf node of the category hierarchy tree T, except for the root node. This leaf node is a special child node of each non-leaf node. The root node in G corresponds to the input feature vector of the MR-head classification head. Each of the remaining non-leaf nodes corresponds to a hidden activation output vector in the MR-head. Each leaf node corresponds to the output logit value of a certain category. Each edge corresponds to a network module, whose input is the feature vector represented by the node at the tail of the edge, and its output is the feature vector represented by the node at the head of the edge. Each edge pointing to a non-leaf node corresponds to a basic fully connected block, which is composed of a concatenated FC layer, GELU layer, LN layer, and Dropout layer. Each edge pointing to a leaf node corresponds to a fully connected layer with an output dimension of 1. The logit value of the category corresponding to a leaf node in T is the output of the leaf node at the same position in G; the logit value of the category corresponding to a non-leaf node in T is the output of the special child node of the non-leaf node at the same position in G. The logit value is ,category The logit value is In hierarchical multi-label classification tasks, the output of the fully connected (FC) layer in the basic fully connected block is the hidden vector of the non-leaf nodes. MR-head to Calculate the mean of each element in the array, and then add the mean to the output corresponding to the special child node of the non-leaf node. superior: in, This is the original logit value, which is the output corresponding to the special child node of the non-leaf node. The hidden vector of the non-leaf nodes. Output dimension for the hidden layer. This is the final logit value; let the category be... The corresponding non-leaf nodes are , Special child nodes are , The hidden vector is ; The P-head classification head, based on the MR-head, further introduces a prior probability of the parent class using the total probability formula to strengthen the relationship between the sub-class probability values ​​and the parent class probability values ​​output by the model. Let the parent class be... Subcategories are Using the law of total probability, the sample Category The probability of can be expressed as follows: in, Indicates sample category The label, the subcategory logit value output by MR-head Logit value of parent category The impact will As , Treat as prior probability of parent category Therefore, in the P-head, the sample Category The probability of is expressed as follows: P-head modifies the formula for calculating the prior probability of the parent class, as shown in the following formula: in, This is a hyperparameter used to adjust the logit value of the parent category. Perform a linear transformation.

Images