A text and image false information detection method and system based on modal fusion

Abstract

The application discloses a kind of text and image false information detection method and system based on modal fusion, belong to network security technical field.The present application is by identifying text in original image and supplementing the deficiency of original text, text features and image features are extracted using pre-training model XLnet, Swin-T, CLIP, then after MMoE processing, different text features and image features of each 2 groups are obtained, text comprehensive features are extracted by each 1 group of text features therein, image comprehensive features are extracted by each 1 group of image features therein, modal fusion features are obtained by the fusion of the remaining each 1 group of text features and image features, the obtained features are spliced and input into discriminator, to detect false information in text and image.The present application can enhance the detection ability of text and image false information, effectively solve the problem that the area of text information in picture is too large or the picture factor has little effect on the discrimination effect in text and image false information.

Description

Technical Field

[0001] This invention belongs to the field of network security technology, and specifically relates to a method and system for detecting fake information in images and text based on modal fusion. Background Technology

[0002] Image and text misinformation detection, as the name suggests, involves analyzing both image and text information to determine the authenticity of the information. The general process is as follows: first, relevant methods are used to extract text modal features and image modal features separately; then, feature fusion is performed using methods such as splicing; finally, a binary classification task is performed based on the fused features to achieve relevant detection.

[0003] Currently, common methods for detecting misinformation in images and text include att-RNN. [1] EANN [2] SpotFake [3] MAVE [4] CARMN [5] While methods like these exist, current image-text misinformation detection methods still have corresponding defects and some problems. Problem one: Most multimodal misinformation detection methods ignore the text information contained within images. In Chinese social media information, some accompanying images have a large area of ​​text information. Problem two: Intermodal consistency is not the only factor determining misinformation. One situation is that the text or image information is forged, making the image and text information consistent. Another situation is, as mentioned in Problem one, where the area of ​​text information in some image modal information is too large, causing the multimodal problem to degenerate into a single-modal text misinformation recognition problem. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for detecting false information in images and text, thereby enhancing the detection capability of false information in images and text and effectively solving the problem that the area of ​​text information in the image is too large or that the image factors are not significant, thus affecting the identification effect.

[0005] The technical solution proposed by this invention to achieve the above objectives is as follows:

[0006] A method for detecting fake information in images and text based on modality fusion includes the following steps:

[0007] For graphic information consisting of original images and original text, identify the text in the original image and combine it with the original text;

[0008] Using the pre-trained XLnet model [6] and CLIP model [8] Extract text features from combined text;

[0009] Using the pre-trained model Swin-T [7] The CLIP model extracts image features from the original images mentioned above;

[0010] The extracted text and image features are used in the MMoE module. [9] The processing yields two sets of XLnet text features, two sets of CLIP text features, two sets of Swin-T image features, and two sets of CLIP image features.

[0011] Textual comprehensive features were extracted based on one set of XLnet text features and one set of CLIP text features;

[0012] Image comprehensive features were extracted based on one set of Swin-T image features and one set of CLIP image features;

[0013] The remaining 1 set of XLnet text features, 1 set of CLIP text features, 1 set of Swin-T image features, and 1 set of CLIP image features are fused together to obtain modality fusion features;

[0014] The text synthesis features, image synthesis features, and modality fusion features are concatenated to obtain the final features;

[0015] The final features are then fed into a discriminator to detect false information in the images and text.

[0016] Furthermore, the identified multiple text segments are first concatenated, and then the concatenated text is segmented using the jieba word segmentation library. The percentage of segments with a length of 1 is counted. If it exceeds 20%, it is considered invalid text; otherwise, it is considered valid text and combined with the original text.

[0017] Furthermore, a text multimodal processing module is used to extract comprehensive text features. This module includes four fully connected layers. One fully connected layer is used to reduce the dimensionality of the XLnet text features to obtain one-dimensional features. After concatenating the one-dimensional features with the CLIP text features, the other three fully connected layers are used to extract the comprehensive text features.

[0018] Furthermore, the image multimodal processing module is used to extract comprehensive image features. This module includes four fully connected layers. One fully connected layer is used to reduce the dimensionality of the Swin-T image features to obtain one-dimensional features. After concatenating the one-dimensional features with the CLIP image features, the other three fully connected layers are used to extract comprehensive image features.

[0019] Furthermore, modal fusion features are obtained using a feature fusion module, which includes a multilayer perceptron (MLP), a co-attention model, a fully connected layer, and an FFN layer. The co-attention model employs a multi-head attention mechanism. The steps for obtaining modal fusion features include:

[0020] The XLnet text features and Swin-T image features are processed through a multilayer perceptron and a Co-Attention model to obtain two sets of modality enhancement features, namely text enhancement features and image enhancement features;

[0021] The two sets of modal enhancement features were dimensionality reduced using a fully connected layer and then concatenated into a single set of one-dimensional modal enhancement features.

[0022] One-dimensional modality enhancement features are extracted using an FFN layer to obtain preliminary fused features;

[0023] The CLIP-Similarity algorithm is used to calculate the image-text similarity index for CLIP text features and CLIP image features;

[0024] The modality fusion features are obtained by multiplying the initial fusion features with the image-text similarity index.

[0025] Furthermore, the discriminator includes three fully connected layers. These three fully connected layers determine the false information in the image and text based on the final features and output the probability value of the false information.

[0026] Furthermore, the final features are obtained by pre-processing an image and text training set with labels for real and fake samples through multiple steps and then input into the discriminator for discriminator training. The binary cross-entropy loss function is calculated to optimize the discriminator parameters. The trained discriminator is then used for formal image and text fake information detection.

[0027] A system for detecting fake information in images and text based on modal fusion includes:

[0028] Image text recognition module, used to recognize text in the original image;

[0029] The preprocessing module is used to concatenate multiple text segments, perform word segmentation, and obtain valid text.

[0030] The XLnet model is used to extract XLnet text features.

[0031] The Swin-T model is used to extract features from Swin-T images.

[0032] CLIP model is used to extract CLIP text features and CLIP image features;

[0033] The MMoE module is used to process XLnet text features, CLIP text features, Swin-T image features, and CLIP image features, resulting in 2 sets of XLnet text features, 2 sets of CLIP text features, 2 sets of Swin-T image features, and 2 sets of CLIP image features.

[0034] The text multimodal processing module is used to extract comprehensive text features based on one set of XLnet text features and one set of CLIP text features;

[0035] The image multimodal processing module is used to extract comprehensive image features based on a set of Swin-T image features and a set of CLIP image features;

[0036] The feature fusion module is used to fuse one set of XLnet text features, one set of CLIP text features, one set of Swin-T image features, and one set of CLIP image features to obtain modality fusion features;

[0037] The discriminator is used to detect false information in text and images based on the final features concatenated from textual comprehensive features, image comprehensive features, and modality fusion features.

[0038] The beneficial effects achieved by this invention are:

[0039] This invention overcomes the shortcomings of existing technologies in detecting image-text misinformation by recognizing the textual information within images. It employs XLnet and Swin-T feature extraction techniques to extract effective textual and image-related features. Furthermore, it utilizes the MMoE module to design a joint discrimination mechanism combining single-modal features and modality fusion features, resolving the modality consistency issue. The preprocessing of image-text data is simple, requiring no manual feature extraction. It automatically learns and selects high-level semantic features based on the input content, without relying on specific prior knowledge or other subtasks, exhibiting excellent generalization capabilities. This invention effectively addresses the problem of excessively large textual areas or weak image-related factors affecting the detection of misinformation, achieving higher detection accuracy compared to existing technologies. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the overall process of detecting misinformation in images and text based on modal fusion, according to an embodiment of the present invention.

[0041] Figure 2 This is a flowchart of the text multimodal processing module and the image multimodal processing module.

[0042] Figure 3 This is a flowchart of the feature fusion model. Detailed Implementation

[0043] To make the various technical features, advantages, or effects of the present invention more apparent and understandable, a detailed description is provided below in conjunction with the accompanying drawings.

[0044] This invention specifically proposes a method and system for detecting false information based on modal fusion, the processing flow of which is as follows: Figure 1 As shown, the specific processing steps are as follows:

[0045] (1) First, the text information in the original image (composed of the original image and original text) is initially extracted using OCR technology through the image text recognition module. After obtaining the text, a preprocessing module is used to concatenate the multiple text segments in a top-to-bottom and left-to-right order according to coordinates. Then, the jieba word segmentation library is used for word segmentation. Finally, the percentage of words with a length of 1 is counted. If it exceeds 20%, it is considered invalid text; otherwise, it is considered valid text. Valid text is added to the existing text information for subsequent processing.

[0046] (2) Using a text-image feature extraction model composed of the pre-trained XLnet model, Swin-T model, and CLIP model (the parameters of these models have been frozen), features are extracted from the text and image information processed in the previous step. The text information is limited to 144 Chinese characters. Finally, four sets of feature vectors are obtained—XLnet text feature vector T. xl The dimension is 144×768; the Swin-T image feature vector V st The dimension is 144×1024; CLIP text feature vector T cp The dimension is 512; CLIP image feature vector V cp The dimension is 512.

[0047] (3) The MMoE module is used to process the four sets of feature vectors obtained, outputting two sets of feature vectors corresponding to each set of vectors. The last dimension of each output feature vector is 512. Specifically, a total of eight sets of feature vectors are obtained: two sets of XLnet text feature vectors T. xl-0 and T xl-1 The dimension is 144×512; 2 sets of Swin-T image feature vectors V st-0 and V st-1 The dimension is 144×512; 2 sets of CLIP text feature vectors T cp-0 and T cp-1 The dimension is 512; 2 sets of CLIP image feature vectors V cp-0 and V cp-1 The dimension is 512.

[0048] (4) Construct a text multimodal processing module to process a set of XLnet text feature vectors Txl-1 and 1 set of CLIP text feature vectors T cp-1 Extract comprehensive text features. This text multimodal processing module contains four fully connected layers, and the processing flow is as follows: Figure 2 As shown, firstly, a fully connected layer is used to reduce the dimensionality of the XLnet text feature vector, transforming it into a one-dimensional feature vector of size 512. Then, this one-dimensional feature vector is concatenated with the CLIP text feature vector to obtain a set of one-dimensional feature vectors of size 1024. Finally, three fully connected layers are used to extract text features from this one-dimensional feature vector, obtaining the comprehensive text feature F. t The dimension is 256.

[0049] (5) Construct an image multimodal processing module to process a set of Swin-T image feature vectors V st-1 and 1 set of CLIP image feature vectors V cp-1 Extract comprehensive image features. The image multimodal processing module contains four fully connected layers, and its processing flow is consistent with that of the text multimodal processing module, such as... Figure 2 As shown, firstly, a fully connected layer is used to reduce the dimensionality of the Swin-T image feature vector, transforming it into a one-dimensional feature vector of size 512. Then, this one-dimensional feature vector is concatenated with the CLIP image feature vector to obtain a set of one-dimensional feature vectors of size 1024. Finally, three fully connected layers are used to extract image features from this one-dimensional feature vector, obtaining the comprehensive image feature F. v The dimension is 256.

[0050] (6) Construct a feature fusion module to process the remaining 4 sets of feature vectors—1 set of XLnet text feature vectors T. xl-0 1 set of CLIP text feature vectors T cp-0 1 set of Swin-T image feature vectors V st-0 and 1 set of CLIP image feature vectors V cp-0 A fusion process is performed to obtain modality fusion features. This feature fusion module includes a multilayer perceptron (MLP), a co-attention model, a fully connected layer, and an open-ended nonlinear generation (FFN) layer. The processing flow is as follows: Figure 3 As shown. First, the XLnet text feature vector and the Swin-T image feature vector are input into the MLP, and the output of the MLP is MLP(T). xl-0 ) and MLP(V st-0 The two sets of feature vectors are then used as input to the Co-Attention model, which employs a multi-head attention mechanism. The head calculation method for the corresponding text features is as follows:

[0051]

[0052] Hv = (h1; h2; ...; h m W O

[0053] Among them, h i For the head of the multi-head attention mechanism, H v W is the output of the multi-head attention mechanism. i q W i k W i v W is the weight parameter. O Let d be a parameter matrix. h =d m / m,d m The dimension is set for the multi-head attention mechanism (set to 512), and m is the number of multi-head attention mechanisms (set to 32).

[0054] The text enhancement feature F obtained through the Co-Attention model vt and image enhancement features F tv The calculation method is as follows:

[0055] F vt = Norm(MLP(T xl-0 )+H v *Norm(MLP(T xl-0 )))

[0056] F tv =Norm(MLP(V st-0 )+H v *Norm(MLP(V st-0 )))

[0057] Norm is the normalization function.

[0058] Then the two sets of modal enhancement feature vectors F are obtained. vt and F tv Dimensionality reduction is performed using a fully connected layer, and a set of one-dimensional modality enhancement feature vectors of size 2048 is obtained by concatenation. Then, an FFN layer is used for feature extraction to obtain a preliminary fused feature vector.

[0059] CLIP text feature vector T cp-0 and CLIP image feature vector V cp-0 The image-text similarity index α is calculated using the CLIP-Similarity algorithm. The CLIP-Similarity algorithm is a method for calculating the similarity between images and text based on the CLIP model. The calculation method is as follows:

[0060]

[0061] Finally, the initial fused feature vector is multiplied by the image-text similarity index α to obtain the final modality fusion feature F. c .

[0062] (7) The text comprehensive features F obtained above t Image comprehensive features F v Modal fusion features F c The concatenation operation yields a set of one-dimensional final feature vectors. These final feature vectors are then input into a discriminator, which, after passing through three fully connected layers, outputs the probability value p of the false information in the image and text. This predicts the classification result of whether the information is true or false, thus completing the authenticity detection of the image and text information.

[0063] This method pre-constructs an image-text training set with labels for both real and fake samples. This training set is then processed through the steps described above to extract the final features, which are then input into the discriminator for training. The binary cross-entropy loss function is used during discriminator training:

[0064]

[0065] Where y is the sample label, 1 for real samples and 0 for fake samples, and the log base is unlimited.

[0066] Experimental test:

[0067] The experimental data used was the weibo16 Chinese dataset, which consisted of 9404 news articles and images crawled from the Sina Weibo platform between May 2012 and June 2016. These articles were verified using Weibo's official rumor-refuting system. The training set contained 3763 genuine articles and images and 3737 fake articles and images, while the test set contained 951 genuine articles and images and 943 fake articles and images.

[0068] First, the image and text data of the aforementioned news article were preprocessed. For the text data, since the XLnet model was used, only the input length needed to be fixed; that is, all sentences longer than the fixed sequence were pruned, and those shorter were padded with zeros. In the experiment, the fixed sequence length was set to 144 characters. Then, the preprocessed image and text data were input into the method of this invention and existing image and text fake information detection models (att-RNN, EANN, SpotFake, MAVE, CARMN), and the classification results in Table 1 were obtained.

[0069] Table 1. Classification Results of False Information in Images and Text

[0070]

[0071] As can be seen from the classification results in Table 1, the method of the present invention has significantly better accuracy, precision, recall, and F1 score than existing image and text fake information detection models in identifying true and false information in news image and text data on Weibo.

[0072] References:

[0073] [1].Jin Z,Cao J,Guo H,et al.Multimodal fusion with recurrent neural networks for rumor detection on microblogs[C] / / Proceedings of the 25th ACMinternational conference on Multimedia.2017:795-816.

[0074] [2].Wang Y,Ma F,Jin Z,et al.Eann:Event adversarial neural networks for multi-modal fake news detection[C] / / Proceedings of the 24th acm sigkddinternational conference on knowledge discovery&data mining.2018:849-857.

[0075] [3].Singhal S,Shah RR,Chakraborty T,et al.Spotfake:A multi-modalframework for fake news detection[C] / / 2019IEEE fifth international conferenceon multimedia big data(BigMM).IEEE,2019:39-47.

[0076] [4].Khattar D, Goud JS, Gupta M, et al.Mvae: Multimodal variationalautoencoder for fake news detection[C] / / The world wide web conference.2019:2915-2921.

[0077] [5].Song C,Ning N,Zhang Y,et al.A multimodal fake news detectionmodel based on crossmodal attention residual and multichannel convolutionalneural networks[J].Information Processing&Management,2021,58(1):102437.

[0078] [6].Yang Z,Dai Z,Yang Y,et al.Xlnet:Generalized autoregressivepretraining for language understanding[J].Advances in neural informationprocessing systems,2019,32.

[0079] [7].Liu Z,Lin Y,Cao Y,et al.Swin transformer:Hierarchical visiontransformer using shifted windows[C] / / Proceedings of the IEEE / CVFinternational conference on computer vision.2021:10012-10022.

[0080] [8].Radford A,Kim J W,Hallacy C,et al.Learning transferable visualmodels from natural language supervision[C] / / International conference onmachine learning.PMLR,2021:8748-8763.

[0081] [9].Ma J, Zhao Z, Yi X, et al.Modeling task relationships in multi-tasklearning with multi-gate mixture-of-experts[C] / / Proceedings of the 24th ACMSIGKDD international conference on knowledge discovery&data mining.2018:1930-1939.

[0082] The above description is merely an embodiment of the present invention and does not limit the scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of protection claimed by the present invention.

Claims

1. A method for detecting fake information in images and text based on modal fusion, characterized in that, Includes the following steps: For graphic information consisting of original images and original text, identify the text in the original image and combine it with the original text; Text features were extracted from combined text using the pre-trained XLnet and CLIP models. Image features were extracted from the original images using the pre-trained Swin-T and CLIP models. The extracted text and image features were processed using the MMoE module to obtain 2 sets of XLnet text features, 2 sets of CLIP text features, 2 sets of Swin-T image features, and 2 sets of CLIP image features. Textual comprehensive features were extracted based on one set of XLnet text features and one set of CLIP text features; Image comprehensive features were extracted based on one set of Swin-T image features and one set of CLIP image features; The remaining 1 set of XLnet text features and 1 set of Swin-T image features are input into the feature fusion module, and then processed by a multilayer perceptron and a Co-Attention model respectively to obtain text enhancement features and image enhancement features; The text enhancement features and image enhancement features are dimensionality reduced using a fully connected layer, and then the dimensionality-reduced text enhancement features and image enhancement features are concatenated into a set of one-dimensional modal enhancement features; The one-dimensional modal enhancement features are extracted using an FFN layer to obtain preliminary fused features; The CLIP-Similarity algorithm was used to calculate the image-text similarity index for the remaining CLIP text features and the remaining CLIP image features. The preliminary fusion features are multiplied by the image-text similarity index to obtain the modality fusion features. The text synthesis features, image synthesis features, and modality fusion features are concatenated to obtain the final features; The final features are then fed into a discriminator to detect false information in the images and text.

2. The method as described in claim 1, characterized in that, First, the identified multiple text segments are concatenated. Then, the concatenated text is segmented using the jieba word segmentation library. The percentage of segments with a length of 1 is counted. If it exceeds 20%, it is considered invalid text; otherwise, it is considered valid text and combined with the original text.

3. The method as described in claim 1, characterized in that, The text multimodal processing module is used to extract the comprehensive text features. This module includes four fully connected layers. One fully connected layer is used to reduce the dimensionality of the XLnet text features to obtain one-dimensional features. After concatenating the one-dimensional features with the CLIP text features, the other three fully connected layers are used to extract the comprehensive text features.

4. The method as described in claim 1, characterized in that, The image multimodal processing module is used to extract comprehensive image features. This module includes four fully connected layers. One fully connected layer is used to reduce the dimensionality of the Swin-T image features to obtain one-dimensional features. After concatenating the one-dimensional features with the CLIP image features, the other three fully connected layers are used to extract the comprehensive image features.

5. The method as described in claim 1, characterized in that, The formulas for calculating text enhancement features and image enhancement features are as follows: in, For XLnet text features, For Swin-T image features, For the head of the multi-head attention mechanism, For the output of the multi-head attention mechanism, For weight parameters, For the parameter matrix, , Dimensions set for multi-head attention mechanisms The number of multi-head attention mechanisms. For the standardized function, Enhance text features, Enhance image features.

6. The method as described in claim 1, characterized in that, The formula for calculating the image-text similarity index is: in, This is an index for image-text similarity. CLIP text features These are CLIP image features.

7. The method as described in claim 1, characterized in that, The discriminator consists of three fully connected layers. These three fully connected layers determine the false information in the image and text based on the final features and output the probability value of the false information.

8. The method as described in claim 1, characterized in that, The image and text training set with labels for real and fake samples is used in advance. After multiple processing steps to obtain the final features, they are input into the discriminator for discriminator training. The binary cross-entropy loss function is calculated to optimize the discriminator parameters. The trained discriminator is used for formal image and text fake information detection.

9. A system for detecting fake information in images and text based on modal fusion, used to implement the method described in any one of claims 1-8, characterized in that, include: Image text recognition module, used to recognize text in the original image; The preprocessing module is used to concatenate multiple text segments, perform word segmentation, and obtain valid text. The XLnet model is used to extract XLnet text features. The Swin-T model is used to extract features from Swin-T images. CLIP model is used to extract CLIP text features and CLIP image features; The MMoE module is used to process XLnet text features, CLIP text features, Swin-T image features, and CLIP image features, resulting in 2 sets of XLnet text features, 2 sets of CLIP text features, 2 sets of Swin-T image features, and 2 sets of CLIP image features. The text multimodal processing module is used to extract comprehensive text features based on one set of XLnet text features and one set of CLIP text features; The image multimodal processing module is used to extract comprehensive image features based on a set of Swin-T image features and a set of CLIP image features; The feature fusion module is used to fuse one set of XLnet text features, one set of CLIP text features, one set of Swin-T image features, and one set of CLIP image features to obtain modality fusion features; The discriminator is used to detect false information in text and images based on the final features concatenated from textual comprehensive features, image comprehensive features, and modality fusion features.

Images