A multimodal fake news detection method, device, and equipment based on large models

Abstract

The present application relates to a multimodal fake news detection method, device, and equipment based on a large model, and in particular to the field of information processing technology. It includes: inputting the multimodal news to be tested into the large model gte‑Qwen2‑1.5B‑instruct, the pre-trained visual model SwinT, and the pre-trained image-text comparison model Clip, respectively, to obtain text features, image features, and text comparison features and image comparison features; calculating the cosine similarity between the normalized text comparison features and the image comparison features to obtain the image-text similarity; using the image-text similarity as the attention matrix threshold to enhance the text features and image features to extract local similarities and differences features; using the image-text similarity as the weight to enhance the text features and image features to extract global similarities and differences features; inputting the local similarities and differences features and the global similarities and differences features into the fake news classifier to obtain the authenticity prediction value of the multimodal news to be tested output by the fake news classifier.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to a multimodal fake news detection method, device, and equipment based on a large model. Background Art

[0002] Fake news is information that is deliberately fabricated or distorted, intended to mislead the audience. In recent years, with the rapid development of social media platforms, the continuous expansion of user scale, and the convenience of information sharing, the spread of fake news has become more serious, posing a severe challenge to the platforms and society. It is worth noting that a large amount of news contains multiple modalities such as text and images. This diversity of modalities not only increases the complexity of manual review of fake news, but also puts higher technical requirements on end-to-end automated detection algorithms.

[0003] Currently, in the field of multimodal fake news detection, information is usually extracted from a single modality first, and then multimodal feature fusion is performed directly. This approach limits the model's in-depth understanding of multimodal news, affecting the accuracy and robustness of multimodal fake news detection. In addition, large models with excellent performance and strong generalization capabilities are not utilized. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a large model-based multimodal fake news detection method, device, and equipment, which can realize end-to-end multimodal fake news automatic detection and improve the accuracy and robustness of multimodal fake news detection.

[0005] In order to achieve the above purpose, the technical solutions provided by the embodiments of the present application are as follows:

[0006] In a first aspect, the present application provides a multimodal fake news detection method based on a large model, the method comprising: inputting the multimodal news to be tested into the large model gte-Qwen2-1.5B-instruct, the pre-trained visual model SwinT and the pre-trained image-text comparison model Clip respectively, obtaining the text features output by the large model gte-Qwen2-1.5B-instruct, the image features output by the pre-trained visual model SwinT, and the text comparison features and image comparison features output by the pre-trained image-text comparison model Clip; calculating the cosine similarity between the normalized text comparison features and the image comparison features to obtain the image-text similarity; using the image-text similarity as the attention matrix threshold to enhance the text features and the image features to extract local similarities and differences features; using the image-text similarity as the weight to enhance the text features and the image features to extract global similarities and differences features; inputting the local similarities and differences features and the global similarities and differences features into the fake news classifier to obtain the authenticity prediction value of the multimodal news to be tested output by the fake news classifier; wherein the fake news classifier is jointly trained based on news samples, corresponding label values ​​and a preset loss function.

[0007] As an optional implementation in the embodiment of the present application, the multimodal news to be tested is respectively input into the large model gte-Qwen2-1.5B-instruct, the pre-trained visual model SwinT and the pre-trained picture-text comparison model Clip, to obtain the text features output by the large model gte-Qwen2-1.5B-instruct, the image features output by the pre-trained visual model SwinT, and the text comparison features and image comparison features output by the pre-trained picture-text comparison model Clip, including: inputting the multimodal news to be tested into the large model gte-Qwen2-1.5B-instruct, so that the large model gte-Qwen2-1.5B-instruct first extracts the dynamic word vector of the multimodal news to be tested, and then reduces the dimension of the dynamic vector to output text features; the multimodal news to be tested is input into the pre-trained visual model SwinT, so that the pre-trained visual model SwinT first extracts the pixel-level features of the multimodal news to be tested, and then reduces the dimension of the pixel-level features to output image features; the multimodal news to be tested is input into the pre-trained image-text contrast model Clip, so that the text encoder of the pre-trained image-text contrast model Clip extracts and outputs text contrast features, and the image encoder of the pre-trained image-text contrast model Clip extracts and outputs image contrast features.

[0008] As an optional implementation in the embodiment of the present application, the image-text similarity is used as the attention matrix threshold to enhance the text features and image features to extract local similarity and difference features, including: calculating the dot product attention score matrix between the text and the image based on the text features and the image features; dividing the dot product attention score matrix based on the image-text similarity to obtain the similarity attention mask matrix and the difference attention mask matrix; calculating the local weighted similarity features based on the dot product attention score matrix, the similarity attention mask matrix, the text features and the image features; calculating the local weighted difference features based on the text features and image features after dimension expansion and the difference attention mask matrix; calculating the local similarity and difference features based on the local weighted similarity features and the local weighted difference features.

[0009] As an optional implementation in an embodiment of the present application, local weighted similarity features are calculated based on the dot product attention score matrix, the similarity attention mask matrix, the text features and the image features, including: replacing the true value in the similarity attention mask matrix based on the dot product attention score matrix to obtain the similarity attention score; multiplying the normalized similarity attention score with the image feature, and then adding it with the text feature to obtain the local similarity feature; applying the similarity attention mask matrix to the local similarity feature to obtain the local masked similarity feature; inputting the local masked similarity feature into the multi-layer perceptron MLP to obtain the local similarity score; multiplying the local similarity score with the local similarity feature to obtain the local weighted similarity feature.

[0010] As an optional implementation in the embodiment of the present application, local weighted distinct features are calculated based on the text features and image features after dimensionality expansion, and the distinct attention mask matrix, including: after expanding the dimensions of the text features and image features, adding them to obtain the joint image and text features; based on the distinct attention mask matrix, selecting local distinct features from the joint image and text features; inputting the local distinct features into the multi-layer perceptron MLP to obtain the local distinct scores; and multiplying the local distinct scores by the local distinct features to obtain the local weighted distinct features.

[0011] As an optional implementation in an embodiment of the present application, local difference and similarity features are calculated based on local weighted similarity features and local weighted difference features, including: connecting local weighted similarity features and local weighted difference features to obtain local weighted difference and similarity features; inputting the local weighted difference and similarity features into a fully connected layer to obtain selection features; performing a difference and similarity fitting auxiliary task based on the selection features; wherein the difference and similarity fitting auxiliary task includes removing a dimension of a preset width in the selection features to obtain a predicted value output of the difference and similarity fitting auxiliary task; and multiplying the normalized selection features by the local weighted difference and similarity features to obtain local difference and similarity features.

[0012] As an optional implementation in the embodiment of the present application, the image-text similarity is used as a weight to enhance the text features and the image features to extract the global similarity and similarity features, including: normalizing the difference between the text features and the image features to obtain the global difference features; normalizing the Schur product of the text features and the image features to obtain the global similarity features; using the image-text similarity as a weight to constrain the global difference features and the global similarity features, and then obtaining the global similarity constrained difference and similarity features through hierarchical normalization; performing dimensionality reduction and nonlinear transformation on the global similarity constrained difference and similarity features to obtain the global similarity and similarity features.

[0013] As an optional implementation in an embodiment of the present application, local difference and similarity features and global difference and similarity features are input into a fake news classifier to obtain a prediction value of the authenticity of the multimodal news to be tested output by the fake news classifier, including: inputting the local difference and similarity features and the global difference and similarity features into the fake news classifier, so that the fake news classifier first connects the local difference and similarity features with the global difference and similarity features, and then performs nonlinear transformation and normalization on the connection results to output the prediction value of the authenticity of the multimodal news to be tested.

[0014] In a second aspect, the present application provides a multimodal fake news detection device based on a large model, the device comprising:

[0015] A multiple feature extraction component is used to input the multimodal news to be tested into the large model gte-Qwen2-1.5B-instruct, the pre-trained visual model SwinT and the pre-trained image-text comparison model Clip, respectively, to obtain the text features output by the large model gte-Qwen2-1.5B-instruct, the image features output by the pre-trained visual model SwinT, and the text comparison features and image comparison features output by the pre-trained image-text comparison model Clip;

[0016] The image-text similarity calculation component is used to calculate the cosine similarity between the normalized text comparison features and the image comparison features to obtain the image-text similarity;

[0017] A local similarity and difference feature enhancement component is used to enhance text features and the image features by using the image-text similarity as an attention matrix threshold to extract local similarity and difference features;

[0018] A global similarity and difference feature enhancement component is used to enhance text features and image features by using image-text similarity as a weight to extract global similarity and difference features;

[0019] A fake news classification component is used to input local difference and similarity features and global difference and similarity features into a fake news classifier to obtain the authenticity prediction value of the multimodal news to be tested output by the fake news classifier; wherein the fake news classifier is obtained by joint training based on news samples, corresponding label values ​​and a preset loss function.

[0020] In a third aspect, the present application provides an electronic device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the large model-based multimodal fake news detection method as described in the first aspect or any one of its optional embodiments.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the multimodal fake news detection method based on a large model as described in the first aspect or any one of its optional embodiments.

[0022] In a fifth aspect, the present application provides a computer program product, comprising: the computer program product comprises a computer program, and when the computer program runs on a computer, the computer implements the multimodal fake news detection method based on a large model as described in the first aspect or any one of its optional embodiments.

[0023] Compared with the prior art, the technical solution provided by the embodiments of the present application has the following advantages:

[0024] The embodiment of the present application provides a multimodal fake news detection method, device, and equipment based on a large model, wherein the method comprises: firstly inputting the multimodal news to be tested into the large model gte-Qwen2-1.5B-instruct, the pre-trained visual model SwinT, and the pre-trained image-text comparison model Clip, respectively, obtaining the text features output by the large model gte-Qwen2-1.5B-instruct, the image features output by the pre-trained visual model SwinT, and the text comparison features and image comparison features output by the pre-trained image-text comparison model Clip, and realizing multimodal fake news detection. Extraction of heavy features; then, calculate the cosine similarity between the normalized text contrast features and the image contrast features to obtain the image-text similarity, use the image-text similarity as the attention matrix threshold to enhance the text features and image features to extract local similarities and differences features; use the image-text similarity as the weight to enhance the text features and image features to extract global similarities and differences features; input the local similarities and differences features and the global similarities and differences features into the fake news classifier to obtain the authenticity prediction value of the multimodal news to be tested output by the fake news classifier; wherein the fake news classifier is jointly trained based on the news samples and the corresponding label values ​​and the preset loss function. In this way, the embodiment of the present application enhances the similarity and difference between text and image from both the global and local levels to obtain local similarities and differences features and global similarities and differences features, so that the fake news classifier can fully and deeply understand the intrinsic correlation and difference between the multimodal content of the multimodal news to be tested, as well as the relationship between the intrinsic correlation and difference and the authenticity of the news, accurately detect fake news, and improve the accuracy and robustness of multimodal fake news detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0027] Figure 1 A schematic diagram of a flow chart of a multimodal fake news detection method based on a large model provided in an embodiment of the present application;

[0028] Figure 2 A schematic diagram of a process for extracting multiple features provided in an embodiment of the present application;

[0029] Figure 3 A schematic diagram of a process for calculating image-text similarity provided in an embodiment of the present application;

[0030] Figure 4 A schematic diagram of a process for extracting local similarities and differences features provided in an embodiment of the present application;

[0031] Figure 5 A schematic diagram of a process for calculating local weighted similarity features provided in an embodiment of the present application;

[0032] Figure 6 A schematic diagram of a process for calculating local weighted different features provided in an embodiment of the present application;

[0033] Figure 7 A schematic diagram of a process for calculating the joint features of graphics and text provided in an embodiment of the present application;

[0034] Figure 8 A schematic diagram of a process for performing a similarity-difference fitting auxiliary task provided in an embodiment of the present application;

[0035] Fig. 9 A schematic diagram of a process for extracting global similarities and differences features provided in an embodiment of the present application;

[0036] Fig. 10A A schematic diagram of the structure of a multimodal fake news detection model based on a large model provided in an embodiment of the present application Figure 1 ;

[0037] Fig. 10B A schematic diagram of the structure of a multimodal fake news detection model based on a large model provided in an embodiment of the present application Figure 2 ;

[0038] Fig.11 A schematic diagram of the structure of a multimodal fake news detection device based on a large model provided in an embodiment of the present application;

[0039] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In the field of multimodal fake news detection, many research works have emerged in recent years. The following are some examples:

[0041] 1. Multimodal fake news detection via progressive fusion networks. A multimodal progressive fusion network (MPFN) was constructed. The researchers extracted multiple levels of image features from the spatial and frequency domains of the image, and then used multiple multi-layer perceptron mixers (Mlp Mixer) to progressively fuse with text features to utilize information at different levels for multimodal fake news detection. In addition to multi-level feature extraction and fusion, there is also work related to cross-modal consistency.

[0042] 2. A multimodal fake news detection method and device, which extracts inconsistent semantics between multiple text features, visual features and multimodal features through a gating mechanism to help the model detect multimodal fake news.

[0043] 3. Cross-modal Contrastive Learning for Multimodal Fake News Detection (COOLANT). This framework uses cross-modal consistency learning as an auxiliary task of image-text contrast learning to help the model align image and text features at a fine-grained level. Then a cross-modal fusion module is used to learn cross-modal correlations from unimodal features to obtain better cross-modal feature representations, thereby improving the performance of multimodal fake news detection.

[0044] 4. To address the consistency problem at the fine-grained level, the consistency learning fine-grained fusion network helps multimodal fake news detection by exploring the consistency and inconsistency of images and texts at the fine-grained level.

[0045] 5. To further verify the effectiveness of feature fusion and contrastive learning, a multimodal fake news detection method (FMC) based on multi-granularity feature fusion and contrastive learning is proposed. This method uses cross-modal alignment and text-visual collaborative attention with the assistance of contrastive learning to achieve multimodal feature fusion of different granularities, helping the model learn more discriminative news feature representations to enhance the ability to identify fake news.

[0046] However, there are still some urgent problems to be solved. There is a lack of a systematic framework to use the extracted similarities and differences features from both the global and local levels to judge the authenticity of news. This limitation restricts the model's in-depth understanding of multimodal news and affects the accuracy and robustness of multimodal fake news detection.

[0047] In order to solve all or part of the above problems, the embodiment of the present application provides a multimodal fake news detection method, device, and equipment based on a large model, wherein the method comprises: firstly inputting the multimodal news to be tested into the large model gte-Qwen2-1.5B-instruct, the pre-trained visual model SwinT, and the pre-trained image-text comparison model Clip, respectively, to obtain the text features output by the large model gte-Qwen2-1.5B-instruct, the image features output by the pre-trained visual model SwinT, and the text comparison features and image comparison features output by the pre-trained image-text comparison model Clip. Compare features to achieve the extraction of multiple features; then, calculate the cosine similarity between the normalized text contrast features and the image contrast features to obtain the image-text similarity, and use the image-text similarity as the attention matrix threshold to enhance the text features and image features to extract local similarities and differences features; use the image-text similarity as a weight to enhance the text features and image features to extract global similarities and differences features; input the local similarities and differences features and the global similarities and differences features into the fake news classifier to obtain the authenticity prediction value of the multimodal news to be tested output by the fake news classifier; wherein the fake news classifier is jointly trained based on news samples, corresponding label values ​​and a preset loss function.

[0048] In this way, the embodiments of the present application enhance the similarities and differences between texts and images at both the global and local levels to obtain local and global similarities and differences features, thereby enabling the fake news classifier to fully and deeply understand the intrinsic correlations and differences between the multimodal contents of the multimodal news to be tested, as well as the relationship between the intrinsic correlations and differences and the authenticity of the news, accurately detect fake news, and improve the accuracy and robustness of multimodal fake news detection.

[0049] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0050] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0051] A multimodal fake news detection method based on a large model provided in the embodiments of the present application can be implemented by electronic devices, including but not limited to personal computers, laptops, tablet computers, smart phones, etc. The operating system of the electronic device may include Android, a mobile operating system (iOS) developed by Apple, an operating system (Windows) developed by Microsoft Corporation of the United States, etc., and the embodiments of the present application are not limited to this. The electronic device can be run alone to implement the present application, or it can be connected to the network and implement the present application through interactive operations with other computer devices in the network. Among them, the network in which the electronic device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN) network, etc.

[0052] It should be noted that the protection scope of the multimodal fake news detection method based on a large model in the embodiment of the present application is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0053] like Figure 1 As shown, Figure 1 The flowchart of a method for detecting multimodal fake news based on a large model according to an embodiment of the present application is shown in FIG. 1 , which can be performed by a multimodal fake news detection device or electronic device based on a large model. The method mainly includes the following steps S101 to S105:

[0054] S101. Input the multimodal news to be tested into the large model gte-Qwen2-1.5B-instruct, the pre-trained visual model SwinT and the pre-trained image-text comparison model Clip respectively, and obtain the text features output by the large model gte-Qwen2-1.5B-instruct, the image features output by the pre-trained visual model SwinT, and the text comparison features and image comparison features output by the pre-trained image-text comparison model Clip.

[0055] S102, calculating the cosine similarity between the normalized text comparison feature and the image comparison feature to obtain the image-text similarity;

[0056] S103, using the image-text similarity as the attention matrix threshold to enhance the text features and image features to extract local similarities and differences features.

[0057] S104: Using the image-text similarity as a weight to enhance text features and image features, so as to extract global similarities and differences features.

[0058] S105. Inputting the local difference and similarity features and the global difference and similarity features into a fake news classifier to obtain a predicted value of the authenticity of the multimodal news to be tested output by the fake news classifier; wherein the fake news classifier is obtained by joint training based on news samples, corresponding label values, and a preset loss function.

[0059] The above embodiment extracts multiple features of the multimodal news to be tested, and then uses the image-text similarity to enhance the similarity and difference between the text and the image from the global level and the local level respectively, to obtain local similarity and difference features and global similarity and difference features, so that the fake news classifier can fully and deeply understand the inherent correlation and difference between the multimodal content of the multimodal news to be tested, as well as the relationship between the inherent correlation and difference and the authenticity of the news, accurately detect fake news, and improve the accuracy and robustness of multimodal fake news detection.

[0060] In the above step S101, the multimodal news to be tested includes but is not limited to modalities such as text, voice, image and video. The multimodal news to be tested can be crawled from social media. In some embodiments, the original news crawled from social media is preprocessed to obtain the multimodal news to be tested, and the preprocessing includes adjusting the text length to a preset length and adjusting the image size to a preset size. Adjusting the text length can be cropping or supplementing, and adjusting the image size can be scaling, center cropping, etc. Exemplarily, the preset length is 200 tokens and 50 tokens. The preset size can be 224*224.

[0061] gte-Qwen2-1.5B-instruct is a large language model based on the self-attention architecture. Launched by the Alibaba team, it is based on Qwen2 training and supports multiple languages ​​and various downstream tasks. Based on the Transformer architecture, it uses technologies such as swiglu activation, attention qkv bias, group query attention, and has an improved word segmenter suitable for multiple natural languages ​​and codes.

[0062] Swin Transformer (SwinT) is a pre-trained vision model based on the Transformer architecture.

[0063] Contrastive Language-Image Pretraining (Clip) is a pre-trained image-text comparison model launched by OpenAI in 2021. Through contrastive learning, text and images are embedded into a common semantic space, so that the representations of relevant text descriptions and image content in this space are close to each other, while irrelevant ones are far away.

[0064] In some embodiments, such as Figure 2As shown, when executing step S101 (inputting the multimodal news to be tested into the large model gte-Qwen2-1.5B-instruct, the pre-trained visual model SwinT and the pre-trained picture-text comparison model Clip respectively, obtaining the text features output by the large model gte-Qwen2-1.5B-instruct, the image features output by the pre-trained visual model SwinT, and the text comparison features and image comparison features output by the pre-trained picture-text comparison model Clip), the following steps S1011 to S1013 are specifically included:

[0065] S1011. Input the multimodal news to be tested into the large model gte-Qwen2-1.5B-instruct, so that the large model gte-Qwen2-1.5B-instruct first extracts the dynamic word vector of the multimodal news to be tested, and then reduces the dimension of the dynamic word vector to output text features.

[0066] Specifically, the large model gte-Qwen2-1.5B-instruct first extracts the dynamic word vector E of the multimodal news to be tested according to the following formula (1):

[0067] E = Qwen(T; Θ Qwen ) (1)

[0068] In formula (1), T is the text contained in the multimodal news to be tested, which can be described as the multimodal news text to be tested. n}, where x i Represents the i-th text unit (token) in the text to be detected, n represents the total number of tokens contained in the text to be detected, and the value range of i is {1,2,3,…,n}. Θ Qwen These are the model parameters of the large model gte-Qwen2-1.5B-instruct.

[0069] Then, the dynamic word vector is mapped to a smaller dimension through a fully connected layer to obtain text features, as shown in formula (2):

[0070] F T-Qwen =W T-Qwen E+b T-Qwen (2)

[0071] In formula (2), F T-Qwen Represents text features; W T-Qwen is the weight in the fully connected layer, b T-Qwen is the bias value in the fully connected layer. For example, the text feature F T-Qwen The dimension is 256.

[0072] Static word embedding models have been widely used in the field of multimodal fake news detection, such as the neural network model word2vec for natural language processing (NLP), the global vector word representation model (glove), etc. However, they cannot understand the contextual meaning of the text in which the words are located, so they perform poorly when dealing with the problem of polysemy. To solve this problem, we chose the large model gte-Qwen2-1.5B-instruct, which is pre-trained on massive data and has excellent performance, to extract dynamic word vectors.

[0073] S1012, input the multimodal news to be tested into the pre-trained visual model SwinT, so that the pre-trained visual model SwinT first extracts the pixel-level features of the multimodal news to be tested, and then reduces the dimension of the pixel-level features to output image features.

[0074] Specifically, the pre-trained visual model SwinT first extracts the pixel-level features P of the multimodal news to be tested according to the following formula (3):

[0075] P = SwinT(V; Θ SwinT ) (3)

[0076] In formula (3), V is the image contained in the multimodal news to be tested, which can be described as the multimodal news image to be tested. V = {p1, p2, p3, ..., p m}, where p j represents the jth image block in the multimodal news image to be tested, the value range of j is {1,2,3,…,m}, and m represents the total number of image blocks contained in the multimodal news image to be tested. Θ SwinT are the model parameters of the pre-trained visual model SwinT.

[0077] Then, the pixel-level features are mapped to a smaller dimension through a fully connected layer to obtain image features, as shown in formula (4):

[0078] F V-SwinT =W V-SwinT P+b V-SwinT (4)

[0079] In formula (4), F V-SwinT Represents image features; W V-SwinT is the weight in the fully connected layer, b V-SwinT is the bias value in the fully connected layer. For example, the image feature F V-swinT The dimension is 256.

[0080] In multimodal fake news detection, Visual Geometry Group (VGG) and Residual Network (ResNet) are often used to extract visual features. However, due to its structure based on convolutional neural networks, it has certain limitations in extracting global information. This application uses the Tiny version of the SwinTransformer model (swin_tiny_patch4_window7_224) as an image feature extractor. The SwinTransformer model has excellent performance in extracting global image information with the help of the attention mechanism. And its hierarchical design is similar to the Convolutional Neural Network (CNN), which can expand the receptive field layer by layer to ensure that local features can also be well extracted.

[0081] S1013, input the multimodal news to be tested into the pre-trained image-text contrast model Clip, so that the text encoder of the pre-trained image-text contrast model Clip outputs text contrast features, and the image encoder of the pre-trained image-text contrast model Clip outputs image contrast features.

[0082] Specifically, the pre-trained image-text contrast model Clip extracts the text contrast feature F according to formula (5): T-Clip , extract the image contrast feature F according to formula (6) V-Clip :

[0083] F T-Clip = Clip(T; Θ encode-text ) (5)

[0084] F V-Clip = Clip(V; Θ encode-image ) (6)

[0085] In formula (5), Θ encode-textIt is a text encoder (Clip TextEncoder) of the pre-trained image-text comparison model Clip. It can be a Chinese Clip (CN-CLIP_ViT-B / 16) text encoder trained based on the Chinese Roberta Whole Word Mask Basic Model (RoBERTa-wwm-Base). RoBERTa-wwm-Base is a pre-trained language model improved from Google's BERT model. Through large-scale unsupervised learning, it learns the contextual relevance of language from massive text data. (CN-CLIP_ViT-B / 16) consists of a text encoder and a visual encoder. The visual encoder can be trained by the Vision Transformer (ViT) Basic Version. Its input image will be divided into 16*16 patches, which is responsible for converting image information into feature vectors for matching and interaction with text features.

[0086] In formula (6), Θ encode-image It is the image encoder (Clip ImageEncoder) of the pre-trained image-text comparison model Clip. It can be ViT-B / 16. The image encoder is combined with other components such as the text encoder to achieve interaction and fusion between images and text, so as to better understand and generate text descriptions related to images.

[0087] For example, the text contrast feature F T-Clip And image contrast feature F V-Clip The dimension is 512.

[0088] The above-mentioned embodiments realize multiple feature extraction, provide a more comprehensive and multi-dimensional feature representation, and enable the model to have a deeper understanding of the multimodal news to be tested. Single feature extraction will cause a significant decline in model performance, while multiple feature extraction reduces this risk. At the same time, it improves the adaptability to news data and the generalization ability of the model. The extracted text features, image features, text contrast features, and image contrast features contain complementary information, which can improve model performance and accuracy.

[0089] In some embodiments, such as Figure 3 As shown, when executing step S102 (calculating the cosine similarity between the normalized text comparison feature and the image comparison feature to obtain the image-text similarity), the following steps S1021 to S1022 are specifically included:

[0090] S1021. Normalize the text contrast features and the image contrast features.

[0091] The normalization process may be an L2 norm normalization process to ensure the numerical stability of the text contrast feature and the image contrast feature in subsequent calculation processes.

[0092] Specifically, according to formula (7), the text contrast feature F T-Cli[ Perform normalization:

[0093]

[0094] According to formula (8), the image contrast feature F V-Clip Perform normalization:

[0095]

[0096] S1022. Calculate the normalized text comparison features Compare features with images The cosine similarity between them is used to get the image-text similarity.

[0097] According to formula (9), the image-text similarity sim is calculated global :

[0098]

[0099] The above embodiment calculates the image-text similarity based on text contrast features and image contrast features, which helps to better understand multimodal content. For multimodal news to be tested, by evaluating the image-text similarity, it is possible to determine whether the content is consistent and relevant, thereby measuring the authenticity of the news. Calculating image-text similarity can promote feature representation learning of text and images. This learning process will prompt the model to mine more representative features in text and images, so that text features and image features are better aligned in the semantic space, further improving the effect of multimodal fusion.

[0100] In some embodiments, such as Figure 4 As shown, when executing step S103 (using the image-text similarity as the attention matrix threshold to enhance the text features and image features to extract local similarities and differences features), the following steps S1030 to S1034 are specifically included:

[0101] S1030. Calculate the dot product attention score matrix between the text and the image based on the text features and the image features.

[0102] Specifically, the dot product attention score matrix Score between text and image is calculated according to formula (10): ii :

[0103]

[0104] In formula (10), t i From the text feature F T-Qwen , represents the i-th token corresponding to the text feature, and the value range of i is {1,2,3,…,n}.j From the image feature F V-SwinT , represents the jth block corresponding to the image feature, and the value range of j is {1,2,3,…,m}.

[0105] The dot product attention score matrix is ​​calculated to establish cross-modal associations. The dot product attention score matrix can measure the correlation between text features and image features. By calculating this matrix, the degree of association between each part of the text and each area in the image can be found. The dot product attention score matrix is ​​obtained through learning and can adaptively establish the association between text and images based on the characteristics of multimodal fake news and the requirements of multimodal fake news detection tasks.

[0106] Calculate the dot product attention score matrix to filter out key information. The dot product attention score matrix can help filter out key information in text and images. When faced with complex text and image data, not all information is useful for multimodal fake news detection tasks. By calculating the dot product attention score matrix, it is possible to highlight information that is closely related to each other and ignore irrelevant parts.

[0107] Calculate the dot product attention score matrix to enhance information focusing ability. The dot product attention score matrix can guide the model to focus limited computing resources and attention on the most relevant parts of the multimodal fake news detection task, avoiding being disturbed by a large amount of irrelevant information.

[0108] S1031. Divide the dot product attention score matrix based on the image-text similarity to obtain a similar attention mask matrix and a different attention mask matrix.

[0109] Specifically, the image-text similarity is used as the dot product attention score matrix Score ij The division threshold is , that is, the image-text similarity is used as the attention matrix threshold. The dot product attention score matrix with a dot product attention score greater than the image-text similarity sim global Set the elements of to true value (True), and set the dot product attention score matrix to the value of the dot product attention score that is less than the image-text similarity sim global The elements of are set to false values ​​(False) to obtain the difference attention mask matrix, indicating that the elements with dot product attention scores greater than in the dot product attention score matrix need to be masked. The difference attention mask matrix is ​​obtained according to formula (11).

[0110] Mask dif-local ={Score ij >sim global} (11)

[0111] The dot product attention score matrix Score ij The midpoint product attention score is less than the image-text similarity simglobal Set the elements of to true values ​​and convert the dot product attention score matrix Score ij The midpoint product attention score is greater than the image-text similarity sim global The elements of are set to false values ​​to obtain the similar attention mask matrix, indicating that the elements with dot product attention scores less than in the dot product attention score matrix need to be masked. The similar attention mask matrix is ​​obtained according to formula (12).

[0112] Mask sim-local ={Score ij ≤sim global} (12)

[0113] The image-text similarity is used as a threshold to divide the part with high similarity between text words and image tiles and the part with low similarity between text words and image tiles in the dot product attention score matrix.

[0114] S1032. Calculate local weighted similarity features based on the dot product attention score matrix, the similarity attention mask matrix, the text features, and the image features.

[0115] Optionally, when executing step S1032, if Figure 5 As shown, the following steps S501 to S505 are included:

[0116] S501, replace the true value in the similar attention mask matrix based on the dot product attention score matrix to obtain the similar attention score. Specifically, the similar attention score Score is calculated according to formula (13) sim :

[0117] Score sim =masked_fill(Score ij ,Mask sim-local ) (13)

[0118] In formula (13), masked_fill means to convert the dot product attention score matrix Score ij In the similar attention mask matrix Mask sim-local The corresponding position is True, use 10 -9 replace.

[0119] S502: multiply the normalized similar attention score by the image feature, and then add it to the text feature to obtain a local similarity feature.

[0120] When executing step S502, first, the similarity attention score is normalized. The similarity attention score Score can be normalized by the softmax function. simNormalize it to get a probability distribution with a sum of 1. Then, multiply the probability distribution by the image feature F V-SwinT , and obtain the image-text similarity feature. Calculate the image-text similarity feature F according to formula (14): VT-wim :

[0121] F VT-sim =softmax(Score sim )·F V-SwinT (14)

[0122] In the above steps, the query comes from the text feature, and the key and value come from the image feature. However, only the part with higher similarity is retained among the similarity scores of the query and key, and the part with lower similarity is a smaller non-zero value. When a smaller value passes through the softmax activation function, it will be close to 0. In this way, multiplying with the image feature ensures that only the part with higher relevance between the image and the text is retained in the obtained image-text similarity feature.

[0123] Furthermore, the image-text similarity feature F VT-sim With the text feature F T-Qwen Add them together to get the local similarity feature. Specifically, the local similarity feature F is calculated according to formula (15): sim-local :

[0124] F sim-local =F VT-sim +F T-Qwen (15)

[0125] S503: Apply the similar attention mask matrix to the local similarity feature to obtain the local mask similarity feature.

[0126] According to formula (16), the similar attention mask matrix Mask sim-local Acting on local similarity feature F sim-local Get the local mask similarity feature F sim-mask .

[0127] F sim-mask =masked_fill(F sim-local ,Mask sim-local ) (16)

[0128] In formula (16), masked_fill means that the similar attention mask matrix Mask sim-local The corresponding position is True and is replaced by 0.

[0129] Because the text features in the local similarity features do not have text features that have undergone similarity constraints, it is necessary to convert the similarity attention mask matrix Mask sim-local, acting on the local similarity feature, and obtaining the local mask similarity feature. The part of the local similarity feature that is not sufficiently related to the image and text is changed to 0, so as to reduce its influence in the subsequent calculation of the local similarity score.

[0130] S504, inputting the local mask similarity feature into a multi-layer perceptron MLP to obtain a local similarity score.

[0131] According to formula (17) (18), the local mask similarity feature F sim-mask The local similarity score Score is obtained through a two-layer multilayer perceptron (Multilayer Perceptron, MLP) sim-local .

[0132] F sim-middle =Tanh(W sim-middle F sim-mask +b sim-middle ) (17)

[0133] Score sim-local =σ(W sim-local F sim-middle +b sim-local ) (18)

[0134] In formula (17), F sim-middle It is an intermediate feature. Tanh is a hyperbolic tangent activation function with an output range between -1 and 1, which is conducive to the modeling of the Sigmoid activation function. sim-middle is the weight matrix in the first linear layer, b sim-middle is the bias value in the first linear layer.

[0135] In formula (18), σ is the Sigmoid activation function, which can map the output between 0 and 1 to obtain the local similarity score Score sim-local .W sim-local is the weight matrix in the second linear layer, b sim-local is the bias value in the second linear layer.

[0136] S505: Multiply the local similarity score by the local similarity feature to obtain a local weighted similarity feature.

[0137] According to formula (19), the local similarity score Score sim-local and local similarity feature F sim-local Multiply them together to get the local weighted similarity feature F w-sim-local .

[0138] F w-sim-local =Score sim-local ·F sim-local (19)

[0139] The above-mentioned embodiment deeply explores the association between text and image in local details by calculating local weighted similarity features. This fine local matching helps to more accurately understand the relationship between text and image. The local weighted similarity features add local information to the fused feature representation. This rich feature representation helps the model to better handle complex multimodal data and improve the performance of the model in the multimodal fake news detection task. When there is some global information (such as background information, etc.) in the text or image that is irrelevant to the multimodal fake news detection task, the local weighted similarity features can effectively reduce the interference of these noises, accurately find valuable local similar parts, and improve the accuracy of similarity judgment.

[0140] S1033. Calculate local weighted difference features based on the text features and image features after dimension expansion and the difference attention mask matrix.

[0141] Optionally, when executing step S1033, if Figure 6 As shown, the following steps S601 to S604 are included:

[0142] S601, after expanding the dimensions of text features and image features, add them together to obtain a joint image-text feature.

[0143] Step S601 includes the following steps S6011 to S6013: Figure 7 As shown:

[0144] S6011. Expand the dimension of the text feature to make it consistent with the number of blocks. Specifically, expand the dimension of the text feature according to formula (20):

[0145] F TH =repeat(unsqueeze(F T-Qwen ,2),n) (20)

[0146] In formula (20), F TH represents the high-dimensional features of text, unsqueeze means adding a dimension at the corresponding position, and repeat means copying the features to the corresponding dimensions. n represents the number of tiles in the image feature.

[0147] S6012: Expand the dimension of the image feature to make it consistent with the number of text units. Specifically, expand the dimension of the image feature according to formula (21):

[0148] F VH =repeat(unsqueeze(F V-SwinT ,1),m) (21)

[0149] In formula (21), F VHrepresents the high-dimensional features of the image, and m represents the number of tokens in the text features.

[0150] S6013, adding the dimensionally expanded text features and image features to obtain the image-text joint feature. Specifically, the image-text joint feature F is calculated according to formula (22): VT-cat :

[0151] F VT-cat =F VH +F TH (twenty two)

[0152] S602: Based on the different attention mask matrix, select local different features from the joint features of image and text.

[0153] The different attention mask matrix calculated by executing step S1031 (dividing the dot product attention score matrix based on the image-text similarity to obtain the similar attention mask matrix and the different attention mask matrix) is used as a Boolean index to select specific elements from the image-text joint features to obtain the local different features. Specifically, the local different features F are obtained according to formula (23): dif-local :

[0154] F dif-local =F VT-cat [Mask dif-local ] (twenty three)

[0155] S603: Input the local difference features into a multi-layer perceptron MLP to obtain a local difference score.

[0156] The process of obtaining the local weighted dissimilarity feature from the local dissimilarity feature is similar to that of calculating the local weighted similarity feature. However, because the local dissimilarity feature does not have the text feature added to it like the local similarity feature, there is no need to use masking again. The local dissimilarity score is obtained directly through a two-layer MLP, and then multiplied with the local dissimilarity feature to obtain the final local weighted dissimilarity feature. Specifically: the local dissimilarity feature F dif-local After a two-layer MLP, the local difference score Score is obtained dif-local As shown in formula (24) (25):

[0157] F dif-middle =Tanh(W dif-middle F dif-local +b dif-middle ) (twenty four)

[0158] Score dif-local =σ(W dif-local F dif-middle +b dif-local ) (25)

[0159] In formula (24), Fdif-middle is the intermediate feature, W dif-middle is the weight matrix in the first linear layer, b dif-middle is the bias value in the first linear layer.

[0160] In formula (25), W dif-local is the weight matrix in the second linear layer, b dif-local is the bias value in the second linear layer.

[0161] S604, local difference score Score dif-local and local distinctive features F dif-local Multiply them together to get the local weighted difference feature F w-dif-local As shown in formula (26):

[0162] F w-dif-local =Score dif-local ·F dif-local (26)

[0163] The above embodiment calculates the local weighted difference features to find the difference between the text and the image in the local area. The local weighted difference features can be used as a signal to detect possible errors. If the local weighted difference features show large differences in certain key parts, this may mean that there is a problem with the correspondence between the text and the image. Calculating the local weighted difference features adds a dimension of difference to the feature representation of the text and the image. This difference dimension can enable the model to understand the relationship between the text and the image more comprehensively.

[0164] Local weighted dissimilarity features can distinguish between real news and fake news, helping the model identify whether there are unreasonable differences between news text and related images, thereby improving classification accuracy. The calculation of local weighted dissimilarity features can also make the model more resistant to noise and interference in the data, thereby performing multimodal processing more robustly. By taking into account the difference information, the model can learn a wider range of relationship patterns when faced with different types of text-image pairs, which helps improve the model's generalization ability on new and unseen data, thereby better completing the multimodal fake news detection task.

[0165] S1034: Calculate local similarity and difference features based on the local weighted similarity features and the local weighted difference features.

[0166] Optionally, when executing step S1034, if Figure 8 As shown, the following steps S801 to S803 are included:

[0167] S801, connect local weighted similarity features F w-sim-local and the local weighted dissimilarity feature F w-dif-local, and obtain the local weighted similarity and difference feature. According to formula (27), the local weighted similarity and difference feature F is calculated. w-DS-local :

[0168] F w-DS-local =concat(F w-sim-local , F w-dif-local ) (27)

[0169] S802: Input the local weighted similarity and difference features into the fully connected layer to obtain the selected features.

[0170] According to formula (28), the selected feature F is calculated choose :

[0171] F choose =W choose ·F w-DS-local +b choose (28)

[0172] In formula (28), W choose is the weight matrix in the fully connected layer, b choose is the bias value in the fully connected layer.

[0173] S803: Perform the similarity-difference fitting auxiliary task based on the selected features.

[0174] The similarity-difference fitting auxiliary task can help the model use local weighted similar features to fit the unique features of real news, and use local weighted dissimilar features to fit the unique features of fake news. The similarity-difference fitting auxiliary tasks include S8031 and S8032:

[0175] S8031. Multiply the normalized selected features by the local weighted similarity and difference features to obtain local similarity and difference features.

[0176] According to formula (29), the local similarity and difference feature F is calculated DS-local :

[0177] F DS-local =softmax(F choose )·F w-DS-local (29)

[0178] The local weighted similarity features and local weighted dissimilarity features are integrated to calculate the local similarity and difference features, which comprehensively describe the relationship between text and image at the local level. Compared with using similarity or dissimilarity features alone, local similarity and difference features can perform more fine-grained analysis. It does not lose the key information of text and image at the local level, and can more accurately reflect how text and image are related to each other in each local area and what differences exist, thereby helping the model to more accurately identify fake news. Local similarity and difference features can be used as a more effective feature for feature fusion in multimodal fake news detection tasks. By incorporating local similarity and difference features into the input or intermediate representation of the model, the model can be better adapted to different multimodal fake news detection tasks. Comprehensive consideration of local similarity and difference features can reduce the one-sidedness of decision-making and provide a richer basis for the model in the multimodal decision-making process. In addition, using local similarity and difference features as new input features or updating existing features can improve the performance of existing models and enable them to better handle complex local relationships in multimodal data.

[0179] S8032. Remove the dimension of preset width in the selected feature to obtain the predicted value output of the similarity-difference fitting auxiliary task.

[0180] The dimension with a feature width of 1 can be removed to obtain the predicted value output of the similarity-difference fitting auxiliary task. Specifically, the predicted value output P of the similarity-difference fitting auxiliary task is obtained according to formula (30): choose :

[0181] P choose =squeeze(softmax(F choose ),1) (30)

[0182] The prediction output of the similarity-dissimilarity fitting auxiliary task is the similarity-dissimilarity fitting probability. During training, the mean square error loss function is used to calculate the loss value.

[0183] The probability of similarity and difference fitting can detect whether there are anomalies between text and images, and serves as a supplementary part of the model output to enhance the model's interpretability and practicality. A lower probability value may mean that there is a discrepancy or mismatch between the text and the image, which helps to detect fake news. A higher probability prediction value usually indicates a better synergy between the text and the image, and a higher authenticity of the news.

[0184] Through the form of similarity and difference fitting probability, the similarity and difference relationship between text and image can be quantified with an accurate value. This quantification method makes the evaluation of text-image relationship more objective and accurate, avoiding the ambiguity of subjective judgment.

[0185] The above embodiment focuses on and extracts the similarity and difference features of images and texts from a local level, uses the image-text similarity as a threshold, divides the parts of text words and blocks with high and low similarity on the image-text similarity matrix, extracts the similarity and difference features between text tokens and local image blocks, and reveals the authenticity of the news. The similarity-difference fitting auxiliary task helps the model judge real news from similar features and judge fake news from different features.

[0186] In some embodiments, such as Fig. 9 As shown, step S104 (using the image-text similarity as a weight to enhance the text features and image features to extract global similarities and differences features) includes the following steps S1040 to S1043:

[0187] S1040: Normalize the difference between the text feature and the image feature to obtain a global difference feature.

[0188] According to formula (31), the global difference feature F is calculated dif-global :

[0189] F dif-global =L2_norm( F T-Qwen -F V-SwinT ) (31)

[0190] In formula (31), L2_norm is L2 norm normalization.

[0191] S1041. Normalize the Schur product of the text feature and the image feature to obtain a global similarity feature.

[0192] According to formula (32), the global similarity feature F is calculated sim-global :

[0193] F sim-glodal = L2_norm(F T-Qwen ⊙F V-SwinT ) (32)

[0194] In formula (32), ⊙ represents the Schur product (Hadamard product). The global similarity feature is calculated by calculating the Hadamard product of the normalized text contrast feature and image contrast feature, and then normalized by the L2 norm.

[0195] S1042, using the image-text similarity as a weight to constrain global dissimilar features and global similarity features, and then obtaining global similarity constrained dissimilarities and similarities features through hierarchical normalization.

[0196] According to formula (33) (34), calculate the global similarity constraint difference feature F S-DS-global .

[0197] F middle1=concat((1-sim global) ·F dif-global ,sim global ·F sim-global ) (33)

[0198] F S-DS-global =LayerNorm(F middle1 ) (34)

[0199] In formula (33), F middle1 is the intermediate variable feature; concat is the connection operation.

[0200] In formula (34), LayerNorm is layer normalization.

[0201] The image-text similarity is used to constrain the global difference features and global similarity features, and then the global similarity constrained difference features are obtained through hierarchical normalization.

[0202] S1043, performing dimensionality reduction and nonlinear transformation on the global similarity constraint similarity and difference features to obtain global similarity and difference features.

[0203] The global similarity constraint is applied to the similarity and difference features F through a linear layer with a LeakyReLU activation function. S-DS-global Dimensionality reduction is performed to reduce the dimension and introduce nonlinear transformation to improve the fitting ability of the model. The global similarity and difference feature F is calculated according to formula (35): DS-global :

[0204] F DS-global =LeakyReLU(W DS-global F S-DS-global +b DS-global ) (35)

[0205] In formula (35), W DS-global is the weight parameter of the linear layer, b DS-global is the bias value of the linear layer.

[0206] The above embodiment uses the image-text similarity as a weight to help weighted fusion of global similarity features and global difference features to obtain global similarity and difference features. It focuses on and extracts the similarity and difference features of images and texts from a global level, that is, starting from the similarities and differences of the overall semantic information of text and images, and helps the model determine whether the news is false through weighted fusion of image-text similarity.

[0207] Step S103 (using the image-text similarity as the attention matrix threshold to enhance the text features and image features to extract local similarities and differences) and step S104 (using the image-text similarity as the weight to enhance the text features and image features to extract global similarities and differences) use the image-text similarity as the threshold or weight as a guide to help modal fusion. Enhance the similarities and differences between text and images from both the global and local levels. Enable the model to more comprehensively understand the relationship between the inherent correlations and differences in multimodal news and the authenticity of news.

[0208] In some embodiments, in step S105 (inputting the local difference and similarity features and the global difference and similarity features into the fake news classifier to obtain the authenticity prediction value of the multimodal news to be tested output by the fake news classifier), the fake news classifier is a two-layer MLP fully connected layer, the first layer has a ReLU activation function, and the second layer has a softmax activation function. Step S105 includes: first connecting the local difference and similarity features and the global difference and similarity features, and then performing nonlinear transformation and normalization processing on the connection results to output the authenticity prediction value of the multimodal news to be tested.

[0209] According to formula (36), local and global similarity and difference features are connected:

[0210] F DS-cross =concat(F DS-global , F DS-local ) (36)

[0211] In formula (36), F DS-cross It is the local and global similarities and differences.

[0212] According to formula (37), the local and global similarity and difference features F DS-cross Perform a nonlinear transformation:

[0213] MLP DS-cross =ReLU(W DS-cross F DS-cross +b DS-cross ) (37)

[0214] In formula (37), MLP DS-cross is the result of the first fully connected layer, ReLU is the activation function, W DS-cross is the weight matrix of the first fully connected layer; b DS-cross is the bias value of the first fully connected layer.

[0215] According to formula (38), the result of the first fully connected layer MLP DS-cross Perform normalization:

[0216] P detection =softmax(W detection MLPDS-cross +b detection ) (38)

[0217] In formula (38), P detection is the output news authenticity prediction value; W detection is the weight matrix of the second fully connected layer with softmax activation function; b detection is the bias value of the second fully connected layer with softmax activation function.

[0218] During training, the cross entropy loss function is used to calculate the loss value.

[0219] The fake news classifier is used to classify fake news. The first fully connected layer learns the complex combination of input features through linear transformation and ReLU activation function, and introduces nonlinear factors to extract more discriminative features. The second fully connected layer converts these features into category probabilities, so that the fake news classifier can output a clear classification result.

[0220] In some embodiments, the similarity-difference fitting auxiliary task and the fake news classifier are jointly trained to continuously optimize the classification performance. A joint training loss function L may be used, and the joint training loss function is shown in formula (39):

[0221] L= L d +λL C (39)

[0222] In formula (39), L d represents the cross entropy loss function, L C represents the mean square error loss function, λ is the threshold parameter, and its value is between 0 and 1.

[0223] The cross entropy loss function is used to measure the difference between the model's prediction of news authenticity and the label value. The cross entropy loss function is shown in formula (40):

[0224]

[0225] In formula (40), y d represents the label value corresponding to the news sample (i.e. the true value of the sample), Represents the predicted value of the multimodal fake news detection module.

[0226] The mean square error loss function helps the model use local weighted similar features to fit the unique features of real news, and use local weighted different features to fit the unique features of fake news. The mean square error loss function is shown in formula (41):

[0227]

[0228] In formula (41), y c ∈{(1,0),(0,1)}. The first dimension of both represents the weight of the local weighted similar features, and the second dimension represents the weight of the local weighted different features.

[0229] In summary, the embodiments of the present application provide a multimodal fake news detection method based on a large model, which realizes end-to-end multimodal fake news detection and can detect fake news efficiently and accurately. Firstly, the multimodal news to be tested is input into the large model gte-Qwen2-1.5B-instruct, the pre-trained visual model SwinT and the pre-trained image-text comparison model Clip respectively, and the text features output by the large model gte-Qwen2-1.5B-instruct, the image features output by the pre-trained visual model SwinT, and the text comparison features and image comparison features output by the pre-trained image-text comparison model Clip are obtained to realize the extraction of multiple features; then, the cosine similarity between the normalized text comparison features and the image comparison features is calculated to obtain the image-text similarity, and the image-text similarity is used as the threshold of the attention matrix to enhance the text features and image features to extract local similarities and differences features; the image-text similarity is used as the weight to enhance the text features and image features to extract global similarities and differences features; the local similarities and differences features and the global similarities and differences features are input into the fake news classifier to obtain the authenticity prediction value of the multimodal news to be tested output by the fake news classifier; wherein the fake news classifier is jointly trained based on the news samples, the corresponding label values ​​and the preset loss function. In this way, the embodiments of the present application enhance the similarities and differences between texts and images at both the global and local levels to obtain local and global similarities and differences features, thereby enabling the fake news classifier to fully and deeply understand the intrinsic correlations and differences between the multimodal contents of the multimodal news to be tested, as well as the relationship between the intrinsic correlations and differences and the authenticity of the news, accurately detect fake news, and improve the accuracy and robustness of multimodal fake news detection.

[0230] like Fig. 10A and Fig. 10B As shown in the figure, the multimodal fake news detection model includes a multiple feature extraction module, a global difference and similarity feature enhancement module, a local difference and similarity feature enhancement module and a fake news classifier.

[0231] The multi-feature extraction module includes: the large model gte-Qwen2-1.5B-instruct, the pre-trained visual model SwinT and the pre-trained image-text comparison model Clip. The pre-trained image-text comparison model Clip includes a text encoder (Clip TextEncoder) and an image encoder (Clip Image Encoder). The multimodal fake news detection model also includes a data preprocessing module, which is used to truncate and segment the multimodal news text to be tested according to the specified length, and to scale, truncate and normalize the multimodal news images to be tested.

[0232] The preprocessed multimodal news to be tested is input into the multiple feature extraction module to obtain the text features, image features, text comparison features and image comparison features output by the multiple feature extraction module. Based on the normalized text comparison features and image comparison features, the image-text similarity is calculated. Then, the text features, image features and image-text similarity are input into the local difference and similarity feature enhancement module to obtain the local difference and similarity features output by the local difference and similarity feature enhancement module, as well as the predicted value output (difference and similarity fitting probability) of the difference and similarity fitting auxiliary task. The text features, image features and image-text similarity are input into the global difference and similarity feature enhancement module to obtain the global difference and similarity features output by the global difference and similarity feature enhancement module. Further, the local difference and similarity features and the global difference and similarity features are input into the fake news classifier to obtain the authenticity prediction value (classification probability) of the multimodal news to be tested output by the fake news classifier. An end-to-end multimodal fake news detection algorithm is implemented.

[0233] In some embodiments, the multiple feature extraction module extracts text features, image features, text contrast features, and image contrast features using multiple pre-trained models.

[0234] In some embodiments, the global similarity and difference feature enhancement module aims to extract global similarity features and global difference features of images and texts, and then assign different weights to the two features through global similarity, so as to improve the performance of the model in the multimodal fake news detection task at a macro level. The similarity and difference features of images and texts are focused on and extracted from a global level, that is, starting from the similarities and differences of the overall semantic information of texts and images, and weighted fusion of image and text similarity is used to help the model determine whether the news is false.

[0235] In some embodiments, the local similarity and difference feature enhancement module is used to extract microscopic similarity and difference features, such as the similarities and differences between text tokens and local image tiles. Use the image-text similarity as a threshold to divide the parts with high and low similarity between text vocabulary and tiles on the image-text similarity matrix, and then perform cross-modal fusion and enhancement of features to help the model better judge fake news. Focus on and extract the similarities and differences between images and texts from a local level, that is, use the image-text similarity as a threshold to extract the similarities and differences between text tokens and local image tiles to reveal the authenticity of the news. And use the similarity and difference fitting auxiliary task to help the model use local weighted similarity features to fit the unique features of real news, and local weighted difference features to fit the unique features of fake news.

[0236] In some embodiments, a similarity-difference fitting auxiliary task is added to help the model to judge real news by local image-text similarity and fake news by local image-text dissimilarity. The similarity-difference fitting auxiliary task and the fake news detector are jointly trained.

[0237] like Fig.11 As shown, Fig.11 A schematic diagram of the structure of a multimodal fake news detection device based on a large model provided in an embodiment of the present application, the device comprising:

[0238] The multiple feature extraction component 1100 is used to input the multimodal news to be tested into the large model gte-Qwen2-1.5B-instruct, the pre-trained visual model SwinT and the pre-trained picture-text comparison model Clip, respectively, to obtain the text features output by the large model gte-Qwen2-1.5B-instruct, the image features output by the pre-trained visual model SwinT, and the text comparison features and image comparison features output by the pre-trained picture-text comparison model Clip;

[0239] The image-text similarity calculation component 1101 is used to calculate the cosine similarity between the normalized text comparison feature and the image comparison feature to obtain the image-text similarity;

[0240] A local similarity and difference feature enhancement component 1102 is used to enhance text features and image features by using image-text similarity as an attention matrix threshold to extract local similarity and difference features;

[0241] A global similarity and difference feature enhancement component 1103 is used to enhance text features and image features by using image-text similarity as a weight to extract global similarity and difference features;

[0242] The fake news classification component 1104 is used to input local difference and similarity features and global difference and similarity features into a fake news classifier to obtain a predicted value of the authenticity of the multimodal news to be tested output by the fake news classifier; wherein the fake news classifier is obtained by joint training based on news samples, corresponding label values ​​and a preset loss function.

[0243] As an optional implementation in the embodiment of the present application, the multiple feature extraction component 1100 is specifically used to: input the multimodal news to be tested into the large model gte-Qwen2-1.5B-instruct, so that the large model gte-Qwen2-1.5B-instruct first extracts the dynamic word vector of the multimodal news to be tested, and then reduces the dimension of the dynamic vector to output text features; input the multimodal news to be tested into the pre-trained visual model SwinT, so that the pre-trained visual model SwinT first extracts the pixel-level features of the multimodal news to be tested, and then reduces the dimension of the pixel-level features to output image features; input the multimodal news to be tested into the pre-trained image-text comparison model Clip, so that the text encoder of the pre-trained image-text comparison model Clip extracts and outputs text comparison features, and the image encoder of the pre-trained image-text comparison model Clip extracts and outputs image comparison features.

[0244] As an optional implementation in the embodiment of the present application, the image-text similarity calculation component is specifically used to: normalize the text comparison features and the image comparison features; calculate the cosine similarity between the normalized text comparison features and the image comparison features to obtain the image-text similarity.

[0245] As an optional implementation in the embodiment of the present application, the local similarity and difference feature enhancement component 1102 includes: an attention matrix calculation unit, which is used to calculate the dot product attention score matrix between the text and the image based on the text feature and the image feature; a mask matrix calculation unit, which is used to divide the dot product attention score matrix based on the image and text similarity to obtain a similar attention mask matrix and a different attention mask matrix; a local weighted similarity feature calculation unit, which is used to calculate the local weighted similarity feature based on the dot product attention score matrix, the similarity attention mask matrix, the text feature and the image feature; a local weighted different feature calculation unit, which is used to calculate the local weighted different feature based on the text feature and the image feature after dimension expansion, and the different attention mask matrix; a local similarity and difference feature calculation unit, which is used to calculate the local similarity and difference feature based on the local weighted similarity feature and the local weighted different feature.

[0246] As an optional implementation in the embodiment of the present application, the local weighted similarity feature calculation unit is specifically used to: replace the true value in the similarity attention mask matrix based on the dot product attention score matrix to obtain the similarity attention score; multiply the normalized similarity attention score with the image feature, and then add it with the text feature to obtain the local similarity feature; apply the similarity attention mask matrix to the local similarity feature to obtain the local masked similarity feature; input the local masked similarity feature into the multi-layer perceptron MLP to obtain the local similarity score; multiply the local similarity score with the local similarity feature to obtain the local weighted similarity feature.

[0247] As an optional implementation in the embodiment of the present application, the local weighted distinct feature calculation unit is specifically used to: after expanding the dimensions of text features and image features, add them to obtain joint image and text features; based on the distinct attention mask matrix, select local distinct features from the joint image and text features; input the local distinct features into the multi-layer perceptron MLP to obtain local distinct scores; multiply the local distinct scores by the local distinct features to obtain local weighted distinct features.

[0248] As an optional implementation in the embodiment of the present application, the local difference and similarity feature calculation unit is specifically used to: connect the local weighted similarity feature and the local weighted difference feature to obtain the local weighted difference and similarity feature; input the local weighted difference and similarity feature into the fully connected layer to obtain the selection feature; perform the difference and similarity fitting auxiliary task based on the selection feature; wherein the difference and similarity fitting auxiliary task includes removing the dimension of a preset width in the selection feature to obtain the predicted value output of the difference and similarity fitting auxiliary task; and multiplying the normalized selection feature with the local weighted difference and similarity feature to obtain the local difference and similarity feature.

[0249] As an optional implementation in the embodiment of the present application, the global difference and similarity feature enhancement component 1103 is specifically used to: normalize the difference between text features and image features to obtain global difference features; normalize the Schur product of text features and image features to obtain global similarity features; use the image and text similarity as a weight to constrain the global difference features and the global similarity features, and then obtain the global similarity constrained difference and similarity features through hierarchical normalization; perform dimensionality reduction and nonlinear transformation on the global similarity constrained difference and similarity features to obtain global difference and similarity features.

[0250] As an optional implementation in the embodiment of the present application, the fake news classification component 1104 is specifically used to: input local difference and similarity features and global difference and similarity features into the fake news classifier, so that the fake news classifier first connects the local difference and similarity features and the global difference and similarity features, and then performs nonlinear transformation and normalization on the connection results to output the authenticity prediction value of the multimodal news to be tested.

[0251] For the specific definition of the multimodal fake news detection device based on a large model, please refer to the definition of the multimodal fake news detection method based on a large model above, which will not be repeated here. Each module in the above-mentioned multimodal fake news detection device based on a large model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0252] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the present application scheme, and does not constitute a limitation on the large model-based multimodal fake news detection device to which the present application scheme is applied. The specific large model-based multimodal fake news detection device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0253] In one embodiment, the multimodal fake news detection method based on a large model provided by the present application can be implemented in the form of a computer program, which can be used in Fig.12 The electronic device shown is running.

[0254] The electronic device includes: a processor 1201 and a memory 1202. The memory 1202 of the electronic device can store various program modules constituting the multimodal fake news detection device based on a large model. The computer program composed of each program module enables the processor to execute the steps of the multimodal fake news detection method based on a large model in each embodiment of the present application described in this specification. And the same technical effect can be achieved, so it will not be repeated here to avoid repetition.

[0255] In one embodiment, the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the multimodal fake news detection method based on a large model in each embodiment of the present application described in this specification are implemented. And the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0256] In one embodiment, the present application provides a computer program product, including: the computer program product includes a computer program, when the computer program is run on a computer, the computer implements the steps in the multimodal fake news detection method based on a large model in each embodiment of the present application described in this specification. And can achieve the same technical effect, to avoid repetition, it will not be repeated here.

[0257] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code.

[0258] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0259] In the present application, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0260] In this application, memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0261] In this application, computer-readable media includes permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0262] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0263] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal fake news detection method based on a large model, characterized in that: include: Input the multimodal news to be tested into the large model gte-Qwen2-1.5B-instruct, the pre-trained visual model SwinT and the pre-trained image-text comparison model Clip respectively, and obtain the text features output by the large model gte-Qwen2-1.5B-instruct, the image features output by the pre-trained visual model SwinT, and the text comparison features and image comparison features output by the pre-trained image-text comparison model Clip; Calculating the normalized cosine similarity between the text comparison feature and the image comparison feature to obtain the image-text similarity; Using the image-text similarity as an attention matrix threshold to enhance the text features and the image features, so as to extract local similarities and differences features; Using the image-text similarity as a weight to enhance the text features and the image features to extract global similarities and differences features; The local difference and similarity features and the global difference and similarity features are input into a fake news classifier to obtain a predicted value of the authenticity of the multimodal news to be tested output by the fake news classifier; wherein the fake news classifier is obtained by joint training based on news samples, corresponding label values ​​and a preset loss function.

2. The method according to claim 1, characterized in that The multimodal news to be tested is respectively input into the large model gte-Qwen2-1.5B-instruct, the pre-trained visual model SwinT and the pre-trained picture-text comparison model Clip, and the text features output by the large model gte-Qwen2-1.5B-instruct, the image features output by the pre-trained visual model SwinT, and the text comparison features and image comparison features output by the pre-trained picture-text comparison model Clip are obtained, including: Input the multimodal news to be tested into the large model gte-Qwen2-1.5B-instruct, so that the large model gte-Qwen2-1.5B-instruct first extracts the dynamic word vector of the multimodal news to be tested, and then reduces the dimension of the dynamic vector to output the text feature; Input the multimodal news to be tested into the pre-trained visual model SwinT, so that the pre-trained visual model SwinT first extracts pixel-level features of the multimodal news to be tested, and then performs dimensionality reduction on the pixel-level features to output the image features; The multimodal news to be tested is input into the pre-trained image-text contrast model Clip, so that the text encoder of the pre-trained image-text contrast model Clip extracts and outputs the text contrast features, and the image encoder of the pre-trained image-text contrast model Clip extracts and outputs the image contrast features.

3. The method according to claim 2, characterized in that The method of using the image-text similarity as an attention matrix threshold to enhance the text features and the image features to extract local similarities and differences features includes: Based on the text features and the image features, calculating a dot product attention score matrix between the text and the image; Dividing the dot product attention score matrix based on the image-text similarity to obtain a similar attention mask matrix and a dissimilar attention mask matrix; Calculating local weighted similarity features based on the dot product attention score matrix, the similarity attention mask matrix, the text features, and the image features; Calculating local weighted difference features based on the text features and the image features after dimension expansion and the difference attention mask matrix; The local similarity and difference feature is calculated based on the local weighted similarity feature and the local weighted difference feature.

4. The method according to claim 3, characterized in that The calculating the local weighted similarity feature based on the dot product attention score matrix, the similarity attention mask matrix, the text feature and the image feature comprises: Replacing the true value in the similarity attention mask matrix based on the dot product attention score matrix to obtain a similarity attention score; Multiplying the normalized similar attention score by the image feature, and then adding the score to the text feature to obtain a local similarity feature; Applying the similar attention mask matrix to the local similarity feature to obtain a local mask similarity feature; Inputting the local mask similarity feature into a multi-layer perceptron MLP to obtain a local similarity score; The local similarity score is multiplied by the local similarity feature to obtain the local weighted similarity feature.

5. The method according to claim 3, characterized in that: The method of calculating the local weighted difference features based on the text features and the image features after dimension expansion and the difference attention mask matrix includes: After expanding the dimensions of the text features and the image features, adding them together to obtain a joint image-text feature; Based on the dissimilarity attention mask matrix, local dissimilarity features are selected from the joint features of the image and text; Inputting the local difference feature into a multi-layer perceptron MLP to obtain a local difference score; The local weighted difference feature is obtained by multiplying the local difference score by the local difference feature.

6. The method according to claim 3, characterized in that The calculating the local similarity and difference features based on the local weighted similarity features and the local weighted difference features includes: Connecting the local weighted similarity feature and the local weighted difference feature to obtain a local weighted difference and similarity feature; Inputting the local weighted similarity and difference features into a fully connected layer to obtain selected features; Performing a similarity-difference fitting auxiliary task based on the selected features; Among them, the difference-similar fitting auxiliary task includes removing the dimension of preset width in the selected feature to obtain the predicted value output of the difference-similar fitting auxiliary task; and multiplying the normalized selected feature with the local weighted difference-similar feature to obtain the local difference-similar feature.

7. The method according to claim 1, characterized in that The step of using the image-text similarity as a weight to enhance the text feature and the image feature to extract global similarities and differences features includes: Normalizing the difference between the text feature and the image feature to obtain a global difference feature; Normalizing the Schur product of the text feature and the image feature to obtain a global similarity feature; The image-text similarity is used as a weight to constrain the global difference feature and the global similarity feature, and then a global similarity constraint difference feature is obtained through hierarchical normalization; The global similarity constraint similarity and difference features are subjected to dimensionality reduction and nonlinear transformation to obtain the global similarity and difference features.

8. The method according to claim 1, characterized in that The step of inputting the local difference and similarity features and the global difference and similarity features into a fake news classifier to obtain the authenticity prediction value of the multimodal news to be tested output by the fake news classifier comprises: The local difference and similarity features and the global difference and similarity features are input into a fake news classifier, so that the fake news classifier first connects the local difference and similarity features with the global difference and similarity features, and then performs nonlinear transformation and normalization on the connection results to output the authenticity prediction value of the multimodal news to be tested.

9. A multimodal fake news detection device based on a large model, characterized in that: include: A multiple feature extraction component is used to input the multimodal news to be tested into the large model gte-Qwen2-1.5B-instruct, the pre-trained visual model SwinT and the pre-trained image-text comparison model Clip, respectively, to obtain the text features output by the large model gte-Qwen2-1.5B-instruct, the image features output by the pre-trained visual model SwinT, and the text comparison features and image comparison features output by the pre-trained image-text comparison model Clip; A picture-text similarity calculation component is used to calculate the cosine similarity between the normalized text comparison feature and the image comparison feature to obtain the picture-text similarity; A local similarity and difference feature enhancement component is used to enhance the text feature and the image feature by taking the image-text similarity as an attention matrix threshold to extract local similarity and difference features; A global similarity and difference feature enhancement component is used to enhance the text feature and the image feature by using the image-text similarity as a weight to extract global similarity and difference features; A fake news classification component is used to input the local difference and similarity features and the global difference and similarity features into a fake news classifier to obtain the authenticity prediction value of the multimodal news to be tested output by the fake news classifier; wherein the fake news classifier is obtained by joint training based on news samples, corresponding label values ​​and a preset loss function.

10. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the large model-based multimodal fake news detection method as described in any one of claims 1 to 10.

Images