Multimodal malicious meme detection method based on face recognition and hierarchical fusion strategy

Through a multimodal detection method based on face recognition and hierarchical fusion strategy, the problem of insufficient cross-modal information interaction in malicious meme map detection is solved, and the detection accuracy and recognition ability of hatred emotions are improved.

CN114419409BActive Publication Date: 2024-11-26DALIAN MARITIME UNIVERSITY
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202210036924.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-12
Publication Date
2024-11-26
Estimated Expiration
2042-01-12

AI Technical Summary

Technical Problem

Traditional models are difficult to fully communicate information across modalities in malicious meme map detection, resulting in insufficient detection accuracy.

Method used

Using a multimodal malicious meme graph detection method based on face recognition and hierarchical fusion strategy, data representation and fusion are represented and fused by obtaining text, image and attribute data, using a bidirectional long and short-term memory network and a feedforward neural network, combining ResNet and YOLO-v5 extracts features, introduces image attributes as bridges, and promotes information exchange between image and text modalities.

Benefits of technology

It significantly improves the accuracy of malicious meme map detection, can better capture the implicit connection between images and text, and enhances the ability to recognize hatred emotions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114419409B_ABST
    Figure CN114419409B_ABST
Patent Text Reader

Abstract

The present invention discloses a multimodal malicious meme detection method based on face recognition and hierarchical fusion strategy, and relates to the technical field of natural language processing. The present invention introduces a new multimodal fusion model and performs multimodal training on images and texts. This multi-feature fusion network first uses ResNet and YOLO‑v5 to extract image features and attribute features, and then uses attribute features and a bidirectional LSTM network to extract text features. While extracting attribute features, a face recognizer is introduced to obtain the race and gender information of the faces in the meme, so that the hateful emotions can be better captured. The features of the three modalities are then reconstructed and fused into a feature vector for prediction. The evaluation results show that the performance of the model in detecting hateful memes has reached a higher level.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a multimodal malicious meme detection method based on face recognition and hierarchical fusion strategy. Background Art

[0002] The Internet is having a huge impact on daily life. The virtual presence on the Internet reflects not only people's personality and beliefs, but also their prejudices and biases. Billions of people interact with various online content every day. Among them, memes usually combine pictures with specific text (usually an expansion or distortion of the content of the picture) to achieve a humorous effect. Memes have been popular for many years, with more than 180 million posted on different social media platforms. Although some of the content is very funny, more and more content is also harmful and hateful, with hateful emotions, which are called "malicious memes." More and more efforts are needed to quickly detect such memes, expand review efforts and make automatic decisions to delete malicious memes to minimize the harm to readers.

[0003] Malicious meme detection can be considered as a typical visual-language (VL) task. Such problems have received a lot of attention in recent years, and significant progress can be seen in important problems such as visual question answering (VQA) and image captioning. Specifically, the multimodal meme detection task is very similar to VQA, which predicts the answer based on the input image and the question.

[0004] Recent VQA methods work using a multimodal fusion framework, which encodes the image and question using corresponding encoders and then fuses them to obtain the final prediction. Typically, a given image is encoded using a convolutional neural network (CNN) based encoder and a sentence is encoded using a recurrent neural network (RNN) based encoder. With the advancement of Transformer networks, many works have incorporated multi-head self-attention mechanisms into their methods, achieving significant performance improvements. The most important part of Transformer is the self-attention mechanism, which transforms the input features into contextual representations with multi-head attention, making it an excellent framework for sharing information between different modalities.

[0005] Despite the similarities between the two tasks, applying methods from VQA to meme detection will face some problems due to its specificity. First, sentences in memes are not like sentences in VQA, which are mainly based on image content rather than questions. Second, memes have more than one modality - each sample in a malicious meme consists of text and image information - one modality may appear non-malicious while the other modality may be malicious (unimodal malice). For example, non-malicious text may be combined with a malicious image. If the image is changed, the malicious meme will become non-malicious, such as Figure 1 In addition, in typical visual language multimodal datasets, there is a direct relationship between images and texts, and traditional multimodal models can better establish connections between texts and images when processing. However, the connection between texts and images in multimodal memes is more implicit. Summary of the invention

[0006] In view of this, the present invention provides a multimodal malicious meme detection method based on face recognition and hierarchical fusion strategy to solve the problem that traditional models cannot fully carry out cross-modal information interaction, so as to improve the accuracy of multimodal malicious meme detection.

[0007] To this end, the present invention provides the following technical solutions:

[0008] On the one hand, the present invention provides a multimodal malicious meme detection method based on face recognition and hierarchical fusion strategy, comprising:

[0009] S1: Obtain data of three modes in the meme to be detected; the three modes are text, image and attribute;

[0010] S2, obtaining an image guidance vector, an attribute guidance vector and a text guidance vector for data representation of the three modal data;

[0011] S3: Use a bidirectional long short-term memory network Bi-LSTM to perform preliminary fusion of the data representations of the three modalities; the attribute guidance vector of the nonlinear transformation is used as the initial state of the Bi-LSTM;

[0012] S4: The data of the three modalities are fused together by using representation fusion and modality fusion, and finally a fused vector is output; the representation fusion includes: reconstructing the feature vector with the help of low-level original vectors and high-level guide vectors from different modalities; the modality fusion includes: converting the feature vector of each modality into a fixed-length form, using a two-layer feedforward neural network to calculate the attention weight of each modality, and using the attention weight to perform weighted averaging of the feature vectors formed by the conversion to obtain a single vector of fixed length;

[0013] S5: Classify the fused vector through a softmax classifier to obtain a prediction result of whether the meme to be detected is a malicious meme.

[0014] Further, obtaining image data in the meme image to be detected includes: extracting image features from the meme image to be detected using a residual network ResNet;

[0015] Acquiring attribute data in the meme to be detected includes: creating race and gender labels by a Fair-Face face classifier; detecting and classifying each face or head in the image by using the Fair-Face face classifier; mapping the race and gender labels back to a bounding box of a person with an accurate face area; using race and gender as attribute features; and at the same time, extracting other objects in the image as attribute features of the image by using a YOLO-v5 object detection model;

[0016] Acquiring text data in the meme graph to be detected includes: extracting text features from the meme graph to be detected using a bidirectional long short-term memory network.

[0017] Furthermore, obtaining the attribute data in the meme graph to be detected includes:

[0018] Among the many attributes extracted by Fair-Face and YOLO-v5, the word embeddings of five attributes are selected as the original attribute vectors. Two of the attributes should be the gender and race of the face in the image. If there is no face information, the five attributes are all attribute features for target detection.

[0019] Furthermore, the image feature representation includes:

[0020] Image guidance vector v image is the feature vector of all regions output by ResNet The average value of

[0021] Where N r is the number of regions;

[0022] Attribute feature representation includes:

[0023] Generate attribute guidance vector by weighted averaging the original attribute vector;

[0024] The original attribute vector e(a i ) Obtain the attention weight a corresponding to the original attribute vector through a two-layer neural network i , used to construct the attribute guidance vector v attr ; as shown below:

[0025] α i =W2·tanh(W1·e(a i)+b1)+b2;

[0026] α i =softmax(α i );

[0027]

[0028] Among them, α i is the i-th image attribute, e is the GloVe embedding operation; W1 and W2 are weight matrices; b1 and b2 are biases; N a is the number of attributes;

[0029] Text feature representation includes:

[0030] The textual guidance vector is the arithmetic mean of the textual representation of the meme output by the bidirectional LSTM network at each time step:

[0031]

[0032] Where L is the length of the meme text.

[0033] Furthermore, the fused vector is classified by a softmax classifier, including:

[0034] A two-layer fully connected neural network is used as the classification layer. The activation functions of the hidden layer and the output layer are element-wise ReLu function and sigmoid function respectively, and the loss function is cross entropy.

[0035] Furthermore, representation fusion is used to fuse the data of the three modalities together, including:

[0036] Will It is represented as the i-th initial vector from mode m, and the weighted average becomes the guiding vector v of mode m. n , where n is text, image or attribute; for each modality m's ith raw vector, three guide weights are calculated based on the guide vectors of different modalities m The final reconstruction weights of the original vector are the average of the normalized guided weights:

[0037]

[0038] α mn =sofatmax(α mn );

[0039]

[0040]

[0041] Where m and n represent any mode respectively; is the guidance weight of the i-th original vector of mode m under the guidance of mode n, α mn Contains all the original vectors of mode m under the guidance of mode n is the final reconstruction weight of the i-th original vector of modality m; L m Is a sequence Length; W mn1 , W mn2 is the weight matrix between mode m and mode n, b mn1 , b mn2 is the deviation between mode m and mode n.

[0042] On the other hand, the present invention also provides a multi-modal malicious meme detection device based on face recognition and hierarchical fusion strategy, comprising:

[0043] A preprocessing unit, used to obtain data of three modes in the meme to be detected; the three modes are text, image and attribute;

[0044] A data representation unit, used for representing the data of the three modalities to obtain an image guidance vector, an attribute guidance vector and a text guidance vector;

[0045] The pre-fusion unit is used to use a bidirectional long short-term memory network Bi-LSTM to perform pre-fusion on the data representation of the three modes; wherein the attribute guide vector of the nonlinear transformation is used as the initial state of the Bi-LSTM;

[0046] A representation and modality fusion unit is used to fuse the data of the three modalities by using representation fusion and modality fusion, and finally output a fused vector; the representation fusion includes: reconstructing the feature vector with the help of low-level original vectors and high-level guide vectors from different modalities; the modality fusion includes: converting the feature vector of each modality m into a fixed-length form, calculating the attention weight of each modality m by using a two-layer feedforward neural network, and using the attention weight to perform weighted averaging of the feature vectors formed by the conversion to obtain a single vector of fixed length;

[0047] The prediction unit is used to classify the fused vector through a softmax classifier to obtain a prediction result of whether the meme to be detected is a malicious meme.

[0048] On the other hand, the present invention also provides a computer-readable storage medium, which stores a computer instruction set. When the computer instruction set is executed by a processor, it implements the multimodal malicious meme detection method based on face recognition and hierarchical fusion strategy as mentioned above.

[0049] Advantages and positive effects of the present invention:

[0050] The present invention introduces a third modality between images and texts, namely, image attributes (five of the objects in the image are selected as attributes of the image after being identified through target recognition). This modality can serve as a bridge between the image and text modalities, promoting information exchange and fusion between the two in the model. Compared with traditional prediction models, it can better explore the implicit connection between images and texts without introducing too much computational burden, greatly improving the accuracy of detection.

[0051] At the same time, the present invention also adds a face recognition program to the target detection program to better identify the gender and racial discrimination of the faces in the meme picture. The face recognition program can use the gender and race of the faces in the meme picture as one of the attributes of the picture. Other detection models focus more on data processing and fusion, without considering the source of maliciousness and concentrated performance. Given that most malicious content is focused on race and gender, this move will inevitably make the model more accurate in detecting maliciousness. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0053] Figure 1 Schematic diagram of a malicious meme in an embodiment of the present invention;

[0054] Figure 2 Schematic diagram of a multimodal fusion model MFFN model in an embodiment of the present invention;

[0055] Figure 3 The present invention is a flowchart of a multimodal malicious meme detection method based on face recognition and hierarchical fusion strategy in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0057] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0058] like Figure 2 As shown, the present invention proposes a new multimodal fusion model MFFN (Multiple Feature Fusion Network) with three modes: image, text and attribute. The image modality uses ResNet as its encoder, and the other modalities share an LSTM model as their encoder. MFFN uses ResNet and YOLO-v5 to extract image features and attribute features, and uses Bi-LSTM (Bi-directional Long Short-Term Memory, full name bidirectional long short-term memory network) to extract text features. Given that race and gender are always the focus of malicious thoughts, a Fair-face face recognizer is added before the attribute feature channel to extract race and gender information in the picture, which can better capture hateful emotions. The model adopts the methods of representation fusion and early fusion to achieve full exchange of information between models, and extracts and fuses three types of features, namely text features, image features and attribute features, in a novel way. The results show that all three types of features contribute to the improvement of model performance, and the model uses a more advanced deep learning model in detecting malicious memes.

[0059] like Figure 3 As shown, a multimodal malicious meme detection method based on face recognition and hierarchical fusion strategy provided in an embodiment of the present invention includes the following specific steps:

[0060] S1: Obtain three modal data in the meme graph to be detected.

[0061] The number of memes on the Internet has reached hundreds of billions. Although memes on the Internet are usually harmless and sometimes funny, they may be malicious due to the specific combination of images and text. When facing such problems, Facebook provides more than 10,000 examples of malicious memes as a dataset and names it the HM dataset. This dataset is selected as the baseline dataset for model training and detection in the embodiments of the present invention.

[0062] In the embodiment of the present invention, for an image, data preprocessing and labeling is first performed.

[0063] Among them, the preprocessing annotation of the image includes the generation of image attribute information. Attribute features indicate that attributes are high-level concepts of images. In the present invention, attributes are the names of items in the image extracted from the image by the target detection program, and these names will be used as attribute labels for this image. The accuracy of the image attribute features will significantly affect the final result of the prediction. Specifically, race and gender labels are created by the Fair-Face face classifier, and each face or head in the image is detected and classified by the Fair-Face face classifier; the labels are mapped back to the bounding box of the person with the accurate face area; unless the face is occluded, the gender and race information of the portrait in the image are finally obtained. In addition, a target detection model is also needed to extract other objects in the image as attributes of the image. YOLO-v5 is the latest target detection model, which will provide the best model performance and can be used to extract other objects in the image as attributes of the image.

[0064] In order to avoid wasting computing resources, among the many attributes extracted by Fair-Face and YOLO-v5, five attributes are selected as the original attribute vector, two of which should be the gender and race of the face in the image. If there is no face information, the five attributes are all attribute labels for target detection. Finally, the five attribute labels are weighted averaged to generate the attribute guidance vector.

[0065] S2. Perform data representation of three modal data.

[0066] (1) Image feature representation

[0067] The input image I is resized to 448×448 and divided into 14×14 regions into ResNet; each region I is then sent through the ResNet model i (i=1, 2, ..., 196) to obtain the regional feature representation v regioni , that is, the original image vector; the image guide vector v image is the average of all region image vectors:

[0068]

[0069] Where N r is the number of regions, which is 196 in the embodiment of the present invention.

[0070] (2) Attribute feature representation

[0071] Attributes are high-level concepts of images and are considered as an additional modality connecting text and images by directly using the word embeddings of the five predicted attributes of each meme as the raw attribute vector. Then, the attribute predictor is used to predict the five attributes of each image. i (i=1, ..., 5). Generate attribute guidance vector by weighted average. Original attribute vector e(a i ) Obtain the attention weight α corresponding to the attribute through a two-layer neural network i , used to construct the attribute guidance vector v attr . The relevant equation is as follows:

[0072] α i =W2·tanh(W1·e(a i )+b1)+b2;

[0073] α i =softmax(α i );

[0074]

[0075] where α i is the i-th image attribute; e is the GloVe embedding operation (GloVe is a new method for generating word vectors proposed by the Stanford NLP research group in 2014); W1 and W2 are weight matrices; b1 and b2 are biases; N a is the number of attributes, which is set to 5 in the embodiment of the present invention.

[0076] (3) Text feature representation

[0077] Bi-LSTM is used to obtain the representation of the meme text, the text guide vector v text is the hidden state h at each time step t The arithmetic mean is:

[0078]

[0079] Where L is the length of the meme text.

[0080] S3: Preliminary fusion of the three modal data.

[0081] Typically, when performing text classification tasks, the Bi-LSTM initial state will be set to zero, but it is a potential place to inject multimodal information to promote modal understanding of text. In the model proposed in this invention, the attribute guidance vector of the nonlinear transformation is used as the initial state of the Bi-LSTM:

[0082] [h f0 ;c f0 ;h b0 ;c b0 ]=ReLu(W·v attr + b);

[0083] where h f0 ,c f0 is the initial state of the forward LSTM, h b0 ,c b0 is the backward LSTM initial state; [;] is the vector concatenation; ReLu represents the element-wise application of the Rectified Linear Units activation function; W and b are the weight matrix and bias.

[0084] S4: Representation fusion and modality fusion are used to fuse the data of the three modalities together, and finally a fused vector is output.

[0085] Among them, inspired by the attention mechanism in VQA tasks, representation fusion aims to reconstruct feature vectors with the help of low-level original vectors and high-level guided vectors from different modalities.

[0086] Will is represented as the i-th initial vector from modality m (which may be text, image or attribute). The key to this stage is to calculate each The weighted average is then used as the new representation of mode m. In order to utilize as much information as possible and more accurately model the relationship between multiple modes, the present invention utilizes information from all three modes - more specifically, the guide vector v n Where n can be text, image or attribute, the weights of the raw vectors are calculated for each mode. For each ith raw vector of modality m, three guided weights are calculated based on the guided vectors of modality n that are different from modality m. The final reconstruction weights of the original vector are the average of the normalized guided weights:

[0087]

[0088] α mn =sofatmax(α mn );

[0089]

[0090]

[0091] Where m and n represent any mode respectively; is the guidance weight of the i-th original vector of mode m under the guidance of mode n, α mn Contains all the original vectors of mode m under the guidance of mode n is the final reconstruction weight of the i-th original vector of modality m; L m Is a sequence Length; W mn1 , W mn2 is the weight matrix between mode m and mode n, b mn1 , b mn2 is the deviation between mode m and mode n.

[0092] After the fusion of representation, v, which was previously represented as the guidance vector image 、v text 、v attr It is now considered as a feature vector for each modality and is ready to be used as input to the next layer.

[0093] The modality fusion used in the embodiment of the present invention is not to simply connect the feature vectors from different modalities to form a longer vector, but to perform modality fusion under the impetus of the work of: first, the feature vector of each modality m is converted into a fixed-length form, a two-layer feedforward neural network is used to calculate the attention weight of each modality m, and then it is used for the weighted average of the feature vectors formed by the conversion, resulting in the use of a single vector of fixed length. Specifically:

[0094] The eigenvector of each mode m is denoted as v m , is first converted to a fixed-length form v 0 m A two-layer feedforward neural network is implemented to compute the attention weights for each modality m, which are then used for the transformed feature vector v 0 m The result is a single fixed-length vector V fused .

[0095]

[0096]

[0097] v′ m =tanh(W m3 ·v m +b m3

[0098]

[0099] where m is one of the three modes, and is included The vector of m1 , W m2 , W m3 is the weight matrix; b m1 , b m2 , b m3 is the deviation; v fused Denotes the reconstructed feature vector during the fusion process.

[0100] S5: The fused data is classified through a softmax classifier to obtain a prediction result of whether the meme to be detected is a malicious meme.

[0101] In the specific implementation, a two-layer fully connected neural network is used as the classification layer, the activation functions of the hidden layer and the output layer are element-wise ReLu function and sigmoid function respectively, and the loss function is cross entropy.

[0102] For ease of understanding, the following is a specific example of a multi-modal malicious meme detection method based on face recognition and layered fusion strategy in an embodiment of the present invention. Figure 2 As shown in the figure, the meme is first input into MFFN, and the text mode of the meme is obtained by GloVe embedding in MFFN, and the text model is processed by Bi-LSTM to obtain the text feature representation; the image mode of the meme is obtained by ResNet, and then the image feature representation is obtained; the image attribute mode (sausage, meat, knife, Arab, male) of the meme is obtained by Fair-Face face recognizer and Yolov5, and then the image attribute feature representation is obtained; the three feature representations are fused based on the original vectors and guide vectors of the three features by representation fusion, and the reconstructed feature vectors of the three features are obtained respectively, and then the three reconstructed feature vectors are fused by modality fusion to obtain the fused vector, and finally the fused vector is used for classification prediction to obtain the prediction result of whether the input meme is a malicious meme.

[0103] Corresponding to the multimodal malicious meme detection method based on face recognition and layered fusion strategy in the present invention, the present invention also provides a multimodal malicious meme detection device based on face recognition and layered fusion strategy, the device comprising:

[0104] A preprocessing unit, used to obtain data of three modes in the meme to be detected; the three modes are text, image and attribute;

[0105] A data representation unit, used for representing the data of the three modalities to obtain an image guidance vector, an attribute guidance vector and a text guidance vector;

[0106] The pre-fusion unit is used to use a bidirectional long short-term memory network Bi-LSTM to perform pre-fusion on the data representation of the three modes; wherein the attribute guide vector of the nonlinear transformation is used as the initial state of the Bi-LSTM;

[0107] A representation and modality fusion unit is used to fuse the data of the three modalities by using representation fusion and modality fusion, and finally output a fused vector; the representation fusion includes: reconstructing the feature vector with the help of low-level original vectors and high-level guide vectors from different modalities; the modality fusion includes: converting the feature vector of each modality m into a fixed-length form, calculating the attention weight of each modality m by using a two-layer feedforward neural network, and using the attention weight to perform weighted averaging of the feature vectors formed by the conversion to obtain a single vector of fixed length;

[0108] The prediction unit is used to classify the fused vector through a softmax classifier to obtain a prediction result of whether the meme to be detected is a malicious meme.

[0109] For a multimodal malicious meme detection device based on face recognition and layered fusion strategy in an embodiment of the present invention, since it corresponds to a multimodal malicious meme detection method based on face recognition and layered fusion strategy in the above embodiment, the description is relatively simple. For relevant similarities, please refer to the description of a multimodal malicious meme detection method based on face recognition and layered fusion strategy in the above embodiment, which will not be described in detail here.

[0110] An embodiment of the present invention also discloses a computer-readable storage medium, which stores a computer instruction set. When the computer instruction set is executed by a processor, it implements a multimodal malicious meme detection method based on face recognition and hierarchical fusion strategy as provided in any of the above embodiments.

[0111] In the several embodiments provided by the present invention, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0112] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0113] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0114] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal malicious meme detection method based on face recognition and hierarchical fusion strategy, characterized in that: include: S1: Obtain data of three modes in the meme to be detected; the three modes are text, image and attribute; S2, obtaining an image guidance vector, an attribute guidance vector and a text guidance vector for data representation of the three modal data; S3: Use a bidirectional long short-term memory network Bi-LSTM to perform preliminary fusion of the data representations of the three modalities; the attribute guidance vector of the nonlinear transformation is used as the initial state of the Bi-LSTM; S4: The data of the three modalities are fused together by using representation fusion and modality fusion, and finally a fused vector is output; the representation fusion includes: reconstructing the feature vector with the help of low-level original vectors and high-level guide vectors from different modalities; the modality fusion includes: converting the feature vector of each modality into a fixed-length form, using a two-layer feedforward neural network to calculate the attention weight of each modality, and using the attention weight to perform weighted averaging of the feature vectors formed by the conversion to obtain a single vector of fixed length; Among them, representation fusion is used to fuse the data of three modalities together, including: Will Represented as the i-th initial vector from mode m, the weighted average becomes the guiding vector of mode m , where n is text, image or attribute; for the i-th raw vector of each modality m, three guide weights are calculated based on the guide vectors of different modalities m ; The final reconstruction weights of the original vector are the average of the normalized guide weights: ; ; ; ; Where m and n represent any mode respectively; is the guidance weight of the ith original vector of mode m under the guidance of mode n, Contains all the original vectors of mode m under the guidance of mode n ; is the final reconstruction weight of the i-th original vector of modality m; L m Is a sequence The length of W mn1 , W mn2 is the weight matrix between mode m and mode n, b mn1 , b mn2 is the deviation between mode m and mode n; S5: Classify the fused vector through a softmax classifier to obtain a prediction result of whether the meme to be detected is a malicious meme.

2. According to claim 1, a multimodal malicious meme detection method based on face recognition and hierarchical fusion strategy is characterized in that: Acquiring image data in the meme image to be detected includes: extracting image features from the meme image to be detected using a residual network ResNet; Acquiring attribute data in the meme to be detected includes: creating race and gender labels by a Fair-Face face classifier; detecting and classifying each face or head in the image by using the Fair-Face face classifier; mapping the race and gender labels back to a bounding box of a person with an accurate face area; using race and gender as attribute features; and at the same time, extracting other objects in the image as attribute features of the image by using a YOLO-v5 object detection model; Acquiring text data in the meme graph to be detected includes: extracting text features from the meme graph to be detected using a bidirectional long short-term memory network.

3. According to claim 2, a multimodal malicious meme detection method based on face recognition and hierarchical fusion strategy is characterized in that: Obtaining the attribute data in the meme graph to be detected includes: Among the many attributes extracted by Fair-Face and YOLO-v5, the word embeddings of five attributes are selected as the original attribute vectors. Two of the attributes should be the gender and race of the face in the image. If there is no face information, the five attributes are all attribute features for target detection.

4. According to claim 3, a multimodal malicious meme detection method based on face recognition and hierarchical fusion strategy is characterized in that: Image feature representation includes: Image Guide Vector is the feature vector of all regions output by ResNet The average value of Where N r is the number of regions; Attribute feature representation includes: Generate attribute guidance vector by weighted averaging the original attribute vector; Original attribute vector The attention weight corresponding to the original attribute vector is obtained by a two-layer neural network , used to construct the attribute guide vector ; as shown below: ; ; ; in, It is i image attributes, e is the GloVe embedding operation; and is the weight matrix; and is the deviation; N a is the number of attributes; Text feature representation includes: The textual guidance vector is the arithmetic mean of the textual representation of the meme output by the bidirectional LSTM network at each time step: ; Where L is the length of the meme text.

5. According to claim 1, a multimodal malicious meme detection method based on face recognition and hierarchical fusion strategy is characterized in that: The fused vector is classified by the softmax classifier, including: A two-layer fully connected neural network is used as the classification layer. The activation functions of the hidden layer and the output layer are element-wise ReLu function and sigmoid function respectively, and the loss function is cross entropy.

6. A multimodal malicious meme detection device based on face recognition and hierarchical fusion strategy, characterized in that: include: A preprocessing unit, used to obtain data of three modes in the meme to be detected; the three modes are text, image and attribute; A data representation unit, used for representing the data of the three modalities to obtain an image guidance vector, an attribute guidance vector and a text guidance vector; The pre-fusion unit is used to use a bidirectional long short-term memory network Bi-LSTM to perform pre-fusion on the data representation of the three modes; wherein the attribute guide vector of the nonlinear transformation is used as the initial state of the Bi-LSTM; The representation and modality fusion unit is used to fuse the data of the three modalities by using representation fusion and modality fusion, and finally output a fused vector; the representation fusion includes: reconstructing the feature vector by means of low-level original vectors and high-level guide vectors from different modalities; the modality fusion includes: converting the feature vector of each modality m into a fixed-length form, calculating the attention weight of each modality m by using a two-layer feedforward neural network, and using the attention weight for weighted averaging of the feature vectors formed by the conversion to obtain a single vector of fixed length; wherein, fusing the data of the three modalities by using representation fusion includes: Will Represented as the i-th initial vector from mode m, the weighted average becomes the guiding vector of mode m , where n is text, image or attribute; for the i-th raw vector of each modality m, three guide weights are calculated based on the guide vectors of different modalities m ; The final reconstruction weights of the original vector are the average of the normalized guide weights: ; ; ; ; Where m and n represent any mode respectively; is the guidance weight of the ith original vector of mode m under the guidance of mode n, Contains all the original vectors of mode m under the guidance of mode n ; is the final reconstruction weight of the i-th original vector of modality m; L m Is a sequence The length of W mn1 , W mn2 is the weight matrix between mode m and mode n, b mn1 , b mn2 is the deviation between mode m and mode n; The prediction unit is used to classify the fused vector through a softmax classifier to obtain a prediction result of whether the meme to be detected is a malicious meme.

7. A computer-readable storage medium, which stores a computer instruction set. When the computer instruction set is executed by a processor, it implements the multimodal malicious meme detection method based on face recognition and hierarchical fusion strategy as described in any one of claims 1 to 5.

Citation Information

Patent Citations

  • Method and system for creating and using emojis and other graphic content in instant messaging systems

    CN108353256A

  • Social network abnormal user detection method and device based on heterogeneous graph neural network

    CN112861967A

  • Attention-based multi-channel data fusion network architecture and data processing method

    CN113255530A

  • Method and apparatus for performing structured extraction on text, device and storage medium

    US20210201182A1

  • System and method for multimodal emotion recognition

    US20210342656A1