Multi-modal Hate Meme Detection Method Based on Multi-task Learning Network

Abstract

The present invention discloses a multi-modal hate meme detection method based on a multi-task learning network. A number of hate meme samples are collected, and a multi-task learning network including a multi-modal main task and two single-modal auxiliary tasks is constructed. Among them, the labels of the multi-modal main task are manually calibrated, and the labels of the single-modal auxiliary tasks are generated based on a self-supervised strategy. After the multi-task learning network is trained using the hate meme samples, a sub-network composed of an image processing module, a text feature extraction module, an image feature extraction module, a feature splicing module, a first feature mapping module, and a second feature mapping module is used as the hate meme detection network. The meme image to be detected is input into the hate meme detection network to obtain the detection result. The present invention constructs and trains a multi-task learning network based on self-supervised generation of auxiliary labels, extracts a hate meme detection network therefrom, and improves the accuracy of hate meme detection while shortening the training time and reducing the computing power requirements.

Description

Technical Field

[0001] The present invention belongs to the technical field of hate meme detection. More specifically, it relates to a multi-modal hate meme detection method based on a multi-task learning network. Background Art

[0002] A meme is an element of a cultural or behavioral system that spreads from one person to another through imitation or other non-genetic behaviors. Memes come in various types and forms, including but not limited to images, videos, or posts, and their influence in social media dissemination is also increasing. The most popular form of meme content is the meme with text in the picture, which is also the main type of meme currently studied. Figure 1 It is an example diagram of a meme with text in the picture. A large amount of meme data on the Internet poses a remarkable problem. Sometimes, an ordinary sentence or picture doesn't have any special emotional meaning, but when they are combined, they can become mean. Malicious memes thus emerge. People with aggressive motives use such memes to attack vulnerable groups or target specific groups, causing emotional harm to them. The proliferation of hate memes on the Internet is becoming an increasingly serious problem in modern society. As social giants like Facebook need to delete thousands of hate memes every day to protect their users, accurately identifying hate memes has become an important task.

[0003] Under the huge scale of Internet data, hate memes cannot be solved by manually checking each meme data, so artificial intelligence is needed to help solve the problem of detecting hate memes. Detecting hate memes that combine visual and text elements is a relatively new research direction. Researchers have proposed different sentiment analysis models. Hate is also a kind of emotion, but hate content is not always as easy to distinguish as other emotions. Some hate attacks against specific groups are very implicit, so the performance of these models in this task is not very good. Early research on hate focused more on single-modal hate text detection, that is, classifying hate, abusive, or offensive text against individuals or groups based on elements such as gender, nationality, or sexual orientation. These studies on hate detection are very inspiring. However, to further improve the accuracy of detecting such multi-modal data as hate memes, it is necessary to recognize the problems and challenges in multi-modal learning.

[0004] There are five core challenges in multi-modal learning: alignment, translation, feature, fusion, and co-learning, among which feature learning is in the primary position. Multi-task learning can maximize the utilization of information from each modality in multi-modal data, learning multi-modal features in the main task and single-modal features in the auxiliary learning. This method will enhance the model's feature learning ability. Therefore, the present invention designs to incorporate the method of multi-task learning into hate meme detection, and it is found that no one has added the method of multi-task learning to the task of detecting hate memes. In multi-task learning, according to the different guiding methods in feature learning, the existing methods can be divided into forward guidance and reverse guidance. Due to the unification of multi-modal label information, forward guidance has difficulties in capturing modality-specific information. Reverse guidance applies an additional loss function as a constraint condition to guide the multi-modal features to contain consistency and complementarity. In the research work of multi-task networks, independent single-modal annotations need to be introduced, but manual annotation is time-consuming and laborious, causing a great obstacle to the actual implementation of technical solutions. Summary of the Invention

[0005] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a multi-modal hate meme detection method based on a multi-task learning network, constructing and training a multi-task learning network based on self-supervised generated auxiliary labels, extracting a hate meme detection network from the trained multi-task learning network, and improving the accuracy of hate meme detection while shortening the training time and reducing the computing power requirements.

[0006] To achieve the above invention purpose, the multi-modal hate meme detection method based on a multi-task learning network of the present invention includes the following steps:

[0007] S1: Collect a number of hate meme samples. Each meme image in the hate meme sample is an image with text. Annotate a hate meme label for each image. When the label is 1, it means there is hate in the meme image; when the label is 0, it means there is no hate in the meme image.

[0008] S2: Construct a multi-task learning network, including a meme image preprocessing module, a text feature extraction module, an image feature extraction module, a feature splicing module, a first feature mapping module, a second feature mapping module, a third feature mapping module, a fourth feature mapping module, a text single-modal label generation module, a fifth feature mapping module, a sixth feature mapping module, and an image single-modal label generation module, where:

[0009] The meme image preprocessing module is used to extract text T from the input meme image and send it to the text feature extraction module, and then remove the extracted text from the meme image to obtain image V and send it to the image feature extraction module;

[0010] The text feature extraction module is used to extract the text feature F of text T t, and then send it to the feature splicing module and the third feature mapping module;

[0011] The image feature extraction module is used to extract the text feature F of the image V v , and then send it to the feature splicing module and the fifth feature mapping module;

[0012] The feature splicing module is used to splice the text feature F t and the image feature F v to obtain the spliced feature F m and send it to the first feature mapping module;

[0013] The first feature mapping module is used to map the feature F m to an M-dimensional feature vector The value of M is determined according to actual needs. Send the M-dimensional feature vector to the second feature mapping module, the text unimodal label generation module, and the image unimodal label generation module;

[0014] The second feature mapping module is used to map the feature vector to the predicted label of the input meme image

[0015] The third feature mapping module is used to map the text feature F t to an M-dimensional text feature vector F t * , and send it to the fourth feature mapping module and the text unimodal label generation module;

[0016] The fourth feature mapping module is used to map the text feature vector F t * to the predicted label obtained from the text

[0017] The text unimodal label generation module is used to generate the text auxiliary label y m based on the true label y of the input meme image , the feature vector t * and the text feature vector F t ;

[0018] The fifth feature mapping module is used to map the image feature F v to an M-dimensional image feature vector and send it to the sixth feature mapping module and the image unimodal label generation module;

[0019] The sixth feature mapping module is used to map the image feature vector to the predicted label obtained from the image

[0020] The image unimodal label generation module is used to generate an image auxiliary label y according to the true label y of the input meme image m and the feature vector and the image feature vector v ;

[0021] S3: Use the meme image in each hate meme sample collected in step S1 as the input of the multi-task learning network, and the corresponding hate meme label as the true label to train the multi-task learning network;

[0022] S4: For the multi-task learning network trained in step S3, use the sub-network composed of the image processing module, text feature extraction module, image feature extraction module, feature splicing module, first feature mapping module and second feature mapping module as the hate meme detection network;

[0023] S5: For the meme image to be detected, input it into the hate meme detection network to obtain the detection result.

[0024] The multi-modal hate meme detection method based on the multi-task learning network of the present invention collects a number of hate meme samples, constructs a multi-task learning network including a multi-modal main task and two unimodal auxiliary tasks, where the label of the multi-modal main task is manually calibrated, and the label of the unimodal auxiliary task is generated based on the self-supervised strategy. After training the multi-task learning network with the hate meme samples, use the sub-network composed of the image processing module, text feature extraction module, image feature extraction module, feature splicing module, first feature mapping module and second feature mapping module as the hate meme detection network. Input the detected meme image into the hate meme detection network to obtain the detection result.

[0025] The present invention introduces a new self-supervised generation auxiliary label multi-task learning strategy based on independent unimodal annotation and advanced modality-specific feature learning. This method does not require manual annotation of unimodal labels, and it can self-supervisedly generate unimodal auxiliary task labels, improving the accuracy of the model detection result while shortening the training time and reducing the computing power requirements. Description of the Drawings

[0026] Figure 1 is an example diagram of a meme with text in the picture;

[0027] Figure 2 is a flowchart of the specific implementation of the multi-modal hate meme detection method based on the multi-task learning network of the present invention;

[0028] Figure 3 is a structural diagram of the multi-task learning network in the present invention;

[0029] Figure 4 It is the training flow chart of the multi-task learning network in the present invention;

[0030] Figure 5 It is an example diagram of the "benign mixing" method for the dataset adopted in this embodiment. Detailed implementation manners

[0031] The following describes the detailed implementation manners of the present invention in conjunction with the accompanying drawings, so that those skilled in the art can better understand the present invention. It should be particularly noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0032] Embodiment

[0033] Figure 2 It is the flow chart of the detailed implementation manners of the multi-modal hate meme detection method based on the multi-task learning network of the present invention. As Figure 2 shown, the specific steps of the multi-modal hate meme detection method based on the multi-task learning network of the present invention include:

[0034] S201: Collect hate meme samples:

[0035] Collect a number of hate meme samples. Each meme image in the hate meme sample is an image with text. Label each meme image with a hate meme label. When the label is 1, it means there is hate in the meme image. When the label is 0, it means there is no hate in the meme image.

[0036] S202: Construct a multi-task learning network:

[0037] For the detection task of hate memes in the present invention, a multi-task learning network that self-supervised generates auxiliary learning labels is designed. In this multi-task learning network, a multi-modal main task and two single-modal auxiliary tasks are constructed to obtain more information through multi-task learning. Figure 3 It is the structure diagram of the multi-task learning network in the present invention. As Figure 3 shown, the multi-task learning network in the present invention includes a meme image preprocessing module, a text feature extraction module, an image feature extraction module, a feature splicing module, a first feature mapping module, a second feature mapping module, a third feature mapping module, a fourth feature mapping module, a text single-modal label generation module, a fifth feature mapping module, a sixth feature mapping module, and an image single-modal label generation module. Next, each module will be described in detail.

[0038] The meme image preprocessing module is used to extract text T from the input meme image and send it to the text feature extraction module, and then remove the extracted text from the meme image to obtain image V and send it to the image feature extraction module.

[0039] The text feature extraction module is used to extract the text feature F of text T t , and then send it to the feature concatenation module and the third feature mapping module. In this embodiment, a pre-trained twelve-layer BERT model is used as the text extraction module, and the text feature extraction expression is as follows:

[0040]

[0041] where represents the parameters of the BERT model.

[0042] The image feature extraction module is used to extract the image feature F of image V v , and then send it to the feature concatenation module and the fifth feature mapping module. In this embodiment, a pre-trained RESNET101 model is used as the image feature extraction module, and the image feature extraction expression is as follows:

[0043]

[0044] where represents the parameters of the RESNET101 model.

[0045] The feature concatenation module is used to concatenate the text feature F t and the image feature F v , and send the concatenated feature F m to the first feature mapping module.

[0046] The first feature mapping module is used to map the feature F m to an M-dimensional feature vector The value of M is determined according to actual needs, and the M-dimensional feature vector is sent to the second feature mapping module, the text unimodal label generation module, and the image unimodal label generation module.

[0047] The second feature mapping module is used to map the feature vector to the predicted label of the input meme image

[0048] The third feature mapping module is used to map the text feature F t to an M-dimensional text feature vector F t * , and send it to the fourth feature mapping module and the text unimodal label generation module.

[0049] The fourth feature mapping module is used to map the text feature vector F t * to the predicted label obtained from the text

[0050] The text unimodal label generation module is used to generate the true label y of the input meme image m , eigenvector and text feature vector F t * Generate text auxiliary label y t , the following expression can be used:

[0051]

[0052] Among them, ULGM t () represents the text unimodal label generation module.

[0053] The fifth feature mapping module is used to transform the image feature F v Mapped to M-dimensional image feature vector And sent to the sixth feature mapping module and the image unimodal label generation module.

[0054] The sixth feature mapping module is used to transform the image feature vector Mapping to the predicted label obtained from the image

[0055] The image unimodal label generation module is used to generate the true label y of the input meme image m , eigenvector and image feature vector Generate image auxiliary label y v , the following expression can be used:

[0056]

[0057] Among them, ULGM v () represents the image unimodal label generation module.

[0058] According to the above description, the multi-task learning network of the present invention includes six feature mapping modules, each of which is used to project the input features into a low-dimensional space. In this embodiment, the feature mapping modules have the same structure, and each feature mapping module includes a linear layer and a ReLU activation layer, where:

[0059] The linear layer is used to linearly map the input feature f to obtain the feature vector f′ and send it to the ReLU activation layer. The linear mapping can be expressed as follows:

[0060] f′=W l T (f)+b l

[0061] Among them, W lrepresents weight, superscript T represents transpose, b l Indicates bias.

[0062] The ReLU activation layer is used to process the received feature vector f′ using the ReLU activation function to obtain the processed feature vector f″. It can be expressed as follows:

[0063] f″=ReLU(f′)

[0064] In the present invention, the multi-task learning network includes one multimodal task and two unimodal tasks, wherein the label of the multimodal task already exists in the dataset, and the auxiliary labels of the two unimodal auxiliary tasks are generated by two unimodal label generation modules to help improve the accuracy of the main task during the training phase. In this embodiment, the unimodal label generation module does not have any parameters, which makes it an independent module and has no effect on the multi-task network. In this embodiment, the two unimodal label generation modules generate auxiliary labels y based on the self-supervised strategy. s , the specific method is as follows:

[0065] The hate class center of the current meme image feature is Non-hate centers are The hate class center of the unimodal feature s is Non-hate centers are

[0066] Calculate the feature vectors of the input meme images respectively and the eigenvector of the unimodal feature s The distance from the corresponding two cluster centers:

[0067]

[0068] Among them, || ||2 means to obtain the two norm, d j is a preset scaling factor used to represent the dimension, j∈{m,s}.

[0069] Calculate the feature vectors of the input meme images respectively and the eigenvector of the single mode s The relative distance α from the hatred center and the non-hatred center j :

[0070]

[0071] Among them, ε is a preset minimum value to avoid zero anomalies.

[0072] Taking into account two aspects at the same time: one aspect is y m with y sRegarding the relationship, on the other hand, to avoid the zero-value problem, the following two equations are obtained:

[0073]

[0074] Then, the auxiliary label y of the unimodal auxiliary task is obtained by equal-weight summation s :

[0075]

[0076] where is the offset value between the auxiliary label y of the unimodal auxiliary task s and the true label of the multimodal main task.

[0077] S203: Train the multi-task learning network:

[0078] Use the meme image in each hate meme sample collected in step S201 as the input of the multi-task learning network, and the corresponding hate meme label as the true label to train the multi-task learning network. The multi-task learning network of the present invention includes a multimodal main task and a unimodal auxiliary task, and a training method is designed specifically. Figure 4 is the training flow chart of the multi-task learning network in this embodiment. As Figure 4 shown, the training steps of the multi-task learning network in this embodiment include:

[0079] S401: Initialize the classification center:

[0080] Randomly initialize the hate class centers and non-hate class centers of each modal feature as

[0081] S402: Let the training round epoch = 1.

[0082] S403: Obtain the output of this round:

[0083] Input the meme images in all hate meme samples into the multi-task learning network to obtain the predicted labels of the i-th meme image in different modalities in this training round and the text auxiliary label image auxiliary label

[0084] S404: Calculate the loss of each modality:

[0085] Calculate the loss of each meme image in each modality respectively

[0086]

[0087] where \(k\in\{m,t,v\}\), represents the true label of the \(i\)-th meme image, and \(loss()\) represents the loss function. In this embodiment, the binary cross-entropy loss function is adopted, and its specific calculation formula is as follows:

[0088]

[0089] S405: Calculate the comprehensive loss and update the parameters:

[0090] The comprehensive loss \(L\) of this training round is calculated using the following formula:

[0091]

[0092] where \(N\) represents the number of hate meme samples, is the weight of the single-modal loss of the \(i\)-th meme image in this training round, and is calculated using the following formula:

[0093]

[0094] where \(tanh()\) is the hyperbolic tangent function, is the single-modal label of the \(i\)-th meme image updated based on momentum in this training round, and is calculated using the following formula:

[0095]

[0096] The single-modal label updated based on momentum in this embodiment is used to calculate the weight of the single-modal loss because the feature vectors of the single modality are dynamically changing, so the generated auxiliary labels may be unstable. Therefore, a momentum update strategy is introduced to control the weight of the single-modal loss.

[0097] Update the parameters in the multi-task learning network according to the calculated comprehensive loss \(L\).

[0098] S406: Determine whether the preset maximum number of training rounds is reached. If so, the training ends; otherwise, go to step S407.

[0099] S407: Update the class centers:

[0100] Calculate the hate class centers and non-hate class centers of each modality feature based on the feature vectors of the meme images in each hate meme sample in this training round. The calculation formulas are as follows:

[0101]

[0102] Among them, I() represents the indicator function, represents the feature vector of modality k in the i-th meme image.

[0103] S408: Let epoch = epoch + 1, and return to step S403.

[0104] S204: Construct a hate meme detection network:

[0105] For the multi-task learning network completed in step S203 training, the sub-network composed of its image processing module, text feature extraction module, image feature extraction module, feature splicing module, first feature mapping module, and second feature mapping module is used as the hate meme detection network.

[0106] S205: Hate meme detection:

[0107] For the meme image to be detected, input it into the hate meme detection network to obtain the detection result.

[0108] To better illustrate the technical effects of the present invention, a specific example is used to experimentally verify the present invention. In the experimental verification of this embodiment, the present invention is compared with the detection performances of hate meme detection methods based on different models.

[0109] The single-modal models include an image classification model and a text classification model. The image classification models are the Image-Grid and Image-Region models. The features of Image-Grid are standard ResNet-152 convolutional features, and the features of Image-Region come from Faster-RCNN with ResNeXt-152 as the backbone. The text classification model is a 12-layer BERT model, which is a bidirectional encoder model of Transformer.

[0110] The multimodal models include Late Fusion, Concat BERT, MMBT-Grid, MMBT-Region, ViLBERT, and VisualBERT. Among them, ViLBERT and VisualBERT are further divided into two versions: unimodal pre-training and multimodal pre-training. Late Fusion is a model that fuses the output averages of the unimodal text model BERT and the unimodal image model ResNet-152 through a simple fusion method. Concat BERT is a model that concatenates the features of the unimodal image model ResNet-152 and the unimodal text model BERT. Both MMBT-Grid and MMBT-Region are supervised multimodal bitransformers models. The former uses Image-Grid features, and the latter uses Image-Region features. Both ViLBERT and VisualBERT are supervised BERT multimodal models, and both of these models have two versions: unimodal pre-training and multimodal pre-training. Among them, ViLBERT and VisualBERT in the table are the unimodal pre-trained ViLBERT and VisualBERT. ViLBERT CC is the ViLBERT pre-trained on the multimodal dataset Conceptual Captions, and VisualBERT COCO is the VisualBERT trained on the multimodal dataset COCO.

[0111] The sample set used in this experimental verification is the hate meme dataset consisting of images and texts released by Facebook AI, which contains more than 10,000 strictly labeled memes with and without hate. Each meme consists of an image and text, where only the unimodal image or text may have hate but the meme itself does not have hate, or the whole meme has hate. This dataset is constructed using some sophisticated processing methods, such as flipping the labels of multimodal hate memes through the "benign mixing" method. Figure 5 It is an example diagram of the "benign mixing" method of the dataset adopted in this embodiment. As Figure 5 shown, the first column are all offensive memes, and the memes are made non-offensive by replacing the images in the second column and the texts in the third column. These processes make it difficult to detect whether the memes have hate using unimodal detection strategies in the dataset, and it is necessary to comprehensively consider the unimodal information of images and texts to infer the results, which increases the difficulty of hate meme detection.

[0112] In this embodiment, through experiments, the accuracy (Accuracy) index of the present invention and various unimodal and multimodal models on the Facebook AI dataset is compared. Table 1 is the comparison table of the detection accuracies of the present invention and the comparative methods in this embodiment.

[0113]

[0114] Table 1

[0115] In Table 1, UP represents unimodal pre-training and MP represents multimodal pre-training. As shown in Table 1, for unimodal models, it can be found that their performance is generally not ideal, which also verifies that the memes in the dataset need to be inferred by combining images and text comprehensively to accurately detect whether they are hateful. Among them, the unimodal text model is better than the unimodal image model, reflecting that if only unimodal is considered, text data contains more information. For multimodal models, it is better than unimodal models, and the degree of fusion affects its performance. The model using the early fusion method is better than the model using the late fusion method. For multimodal pre-training models, it is not difficult to find that the difference between it and unimodal pre-training models is not too large, and there is still room for improvement.

[0116] The present invention uses the late fusion method and unimodal pre-training models. Although the performance of the late fusion method is generally worse than that of the early fusion method, due to the addition of the multi-task learning strategy in the present invention, the detection accuracy is better than that of the advanced early fusion models in the prior art, which also verifies that adding multi-task learning in the hate modality detection task can greatly improve the accuracy of detecting hate modalities. At the same time, in comparison with several multimodal models with similar effects, the present invention also has the characteristics of fast training speed and small computing power consumption, and is more suitable for actual popularization and application.

[0117] Although the above describes the illustrative specific embodiments of the present invention for the convenience of those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concept of the present invention are within the scope of protection.

Claims

1. A multi-modal hate meme detection method based on a multi-task learning network, characterized in that, Including the following steps: S1: Collect a number of hate meme samples. The meme images in each hate meme sample are images with text. Label each image with a hate meme label. When the label is 1, it means there is hate in the meme image. When the label is 0, it means there is no hate in the meme image; S2: Construct a multi-task learning network, including a meme image preprocessing module, a text feature extraction module, an image feature extraction module, a feature splicing module, a first feature mapping module, a second feature mapping module, a third feature mapping module, a fourth feature mapping module, a text unimodal label generation module, a fifth feature mapping module, a sixth feature mapping module, and an image unimodal label generation module, where: The meme image preprocessing module is used to extract the text T from the input meme image and send it to the text feature extraction module, and then remove the extracted text from the meme image to obtain the image V and send it to the image feature extraction module; The text feature extraction module is used to extract the text feature F of text T t , and then send it to the feature splicing module and the third feature mapping module; The image feature extraction module is used to extract the text feature F of the image V v , and then send it to the feature splicing module and the fifth feature mapping module; The feature splicing module is used to splice the text feature F t and the image feature F v to obtain the spliced feature F m and send it to the first feature mapping module; The first feature mapping module is used to map the feature F m into an M-dimensional feature vector The value of M is determined according to actual needs. The M-dimensional feature vector is sent to the second feature mapping module, the text unimodal label generation module, and the image unimodal label generation module; The second feature mapping module is used to map the feature vector to the predicted label of the input meme image The third feature mapping module is used to map the text feature F t into an M-dimensional text feature vector F t * , and send it to the fourth feature mapping module and the text unimodal label generation module; The fourth feature mapping module is used to map the text feature vector F t * to a predicted label obtained from the text The text unimodal label generation module is used to generate the text auxiliary label y according to the true label y of the input meme image m , feature vector and text feature vector F t * to generate the text auxiliary label y t ; The fifth feature mapping module is used to map the image feature F v into an M-dimensional image feature vector and send it to the sixth feature mapping module and the image unimodal label generation module; The sixth feature mapping module is used to map the image feature vector to a predicted label obtained from the image The image single-modal label generation module is used to generate the image auxiliary label y according to the true label y of the input meme image m , feature vector and image feature vector to generate the image auxiliary label y v ; S3: Use the meme images in each hate meme sample collected in step S1 as the input of the multi-task learning network, and the corresponding hate meme labels as the true labels to train the multi-task learning network; S4: For the multi-task learning network trained in step S3, use the sub-network composed of the image processing module, the text feature extraction module, the image feature extraction module, the feature splicing module, the first feature mapping module, and the second feature mapping module as the hate meme detection network; S5: For the meme image to be detected, input it into the hate meme detection network to obtain the detection result.

2. The multimodal hate meme detection method according to claim 1, wherein The feature mapping module in the multi-task learning network includes a linear layer and a ReLU activation layer, where: The linear layer is used to perform a linear mapping on the input feature f to obtain a feature vector f' and send it to the ReLU activation layer; The ReLU activation layer is used to process the received feature vector f' using the ReLU activation function to obtain the processed feature vector f''.

3. The multimodal hate meme detection method according to claim 1, characterized in that, The text unimodal label generation module and the image unimodal label generation module respectively generate auxiliary labels y using the following methods s : Denote the hate class center of the current meme image feature as The non - hate class center is The hate class center of the unimodal feature s is The non - hate class center is s ∈ {t, v}; Calculate the feature vectors of the input meme images respectively and the feature vectors of the unimodal features s to the distances from the corresponding two class centers: where d j is a preset proportionality coefficient, and j ∈ {m, s}; Calculate the feature vectors of the input meme images separately and the feature vectors of the single modality s and the relative distance α between the hate center and the non-hate center j : Where ε is a preset minimum value; The auxiliary label y is calculated using the following formula s :[[]]END]] 4. The multimodal hate meme detection method according to claim 1, wherein The training method of the multi-task learning network is as follows: S3.1: Randomly initialize the hate class centers of each modal feature and the non-hate class centers as k ∈ {m, t, v}; S3.2: Let the training epoch = 1; S3.3: Input the meme images in all hate meme samples into the multi-task learning network to obtain the predicted labels of the i-th meme image in different modalities in this training round and text auxiliary labels image auxiliary labels S3.4: Calculate the losses of each meme image in each modality respectively where \(k\in\{m,t,v\}\), represents the true label of the \(i\)-th meme image, and \(\text{loss}()\) represents the loss function; S3.5: Calculate the comprehensive loss L of this training epoch using the following formula: where N represents the number of hate meme samples, is the weight of the unimodal loss in the i-th meme image in this training round, and is calculated using the following formula: where tanh() is the hyperbolic tangent function, is the unimodal label obtained by momentum update for the i-th meme image in this training round, and is calculated using the following formula: Update the parameters in the multi-task learning network according to the calculated comprehensive loss L; S3.6: Determine whether the preset maximum training epoch is reached. If so, the training ends. Otherwise, go to step S3.7; S3.7: Calculate the hate class centers and non-hate class centers of each modal feature based on the feature vectors of the meme images in each hate meme sample in this training round. And the calculation formulas are as follows: The calculation formulas are as follows: where I() represents the indicator function, represents the feature vector of modality k in the i-th meme image; S3.8: Let epoch = epoch + 1 and return to step S3.

3.

5. The multimodal hate meme detection method according to claim 4, wherein In step S3.4, the loss function uses the binary cross-entropy loss function, and its specific calculation formula is as follows:

Images