A multimodal false information detection method based on semantic consistency

Abstract

The present invention relates to a multimodal false information detection method based on semantic consistency, which aims to solve the problem of semantic interaction between modalities in multimodal false information detection. Specifically, the following steps are included: first, multimodal data of news in social networks are obtained, and text and images are preprocessed. Then, global and local feature extractors for text and images are constructed respectively. Local semantic consistency detection is performed on the extracted local features of the text and the local features of the image, and global semantic consistency detection is performed on the obtained global features, and a joint semantic consistency result is generated by combining the two. The text and image features are multimodally fused using an attention mechanism, and the multimodal features after the fusion of text features and image features and the joint semantic consistency result are comprehensively analyzed to discriminate false information from multiple dimensions and establish a loss function to optimize the model, thereby realizing the detection of false information. The present invention can effectively improve the accuracy of multimodal false information detection and improve the automatic detection of multimodal false information.

Description

Technical Field

[0001] The present invention relates to a multimodal false information detection method based on semantic consistency, belonging to the technical field of false information detection. Background Art

[0002] The rapid development of social networks in recent years has led to an increasing diversity in the ways users express their emotions and opinions, making it increasingly difficult to discern the authenticity of information. Fake news has evolved from traditional plain text to multimodal presentations, incorporating various elements such as visual images. This multimodal false information is not only more engaging but also spreads significantly faster, causing profound harm to the online ecosystem and severely impacting social stability and people's daily lives. Therefore, automated detection of multimodal false news has become a critical and pressing issue in the field of social network analysis, with significant research value and practical applications.

[0003] Multimodal data exhibits complex connections, and the feature relationships within and between modalities are also extremely complex. Traditional feature interaction methods struggle to capture these complex, high-dimensional interactions. For example, there is often a fine-grained, local correspondence between informational text and accompanying images, which exhibits a certain degree of correlation. At the same time, the overall content of the text and image complements each other to convey a more detailed picture of the text and image.

[0004] Most existing multimodal fake news detection methods focus solely on information interactions within modalities or the fusion of global information, often overlooking the relationship between local and global information. While these methods demonstrate a certain degree of simplicity and effectiveness in processing a large number of multimodal fake news cases, they lack in-depth consideration of the interactions between information at different granularities, making it difficult to fully integrate fine-grained and coarse-grained features. In real-world scenarios, fake news often involves subtle manipulation of real image and text information, leading to inconsistencies between local information and overall semantics. Therefore, traditional cross-modal semantic association learning methods, whether globally or locally oriented, struggle to effectively model the semantic interactions between image regions and text words, making it challenging for detection models to reason and judge between heterogeneous modalities.

[0005] To effectively utilize multimodal data and deeply explore the semantic connections between different modalities, it is crucial to detect multimodal disinformation. This requires researchers to explore more complex models that can simultaneously consider the interaction of local and global information, and to improve the accuracy and robustness of fake news detection through multi-level semantic learning. Leveraging advanced technologies such as deep learning, building systems that can capture the deep relationships between multimodal features will provide a powerful tool for combating fake news. Summary of the Invention

[0006] In order to overcome the problem of semantic interaction between modalities in existing multimodal false information detection, the present invention provides a multimodal false information detection method based on semantic consistency. Specifically, it comprises the following steps: first, obtain multimodal data of news in social networks, and preprocess the text and image. Then, construct global and local feature extractors for text and image respectively. Perform local semantic consistency detection on the extracted local features of the text and the local features of the image, perform global semantic consistency detection on the obtained global features, and generate a joint semantic consistency result. Then, use the attention mechanism to assign weights and perform multimodal fusion of text and image features. Finally, combine the fusion result with the semantic consistency result to establish a loss function, thereby realizing the detection of false information. The present invention can effectively improve the accuracy of multimodal false information detection and improve the automatic detection of multimodal false information.

[0007] A multimodal false information detection method based on semantic consistency includes the following steps:

[0008] Step 1: Obtain multimodal news data from social networks and construct a training dataset. Each data in the training dataset includes multimodal news data and a label indicating whether it is false information. The multimodal news data includes text data and image data; perform word segmentation on the text data.

[0009] Step 2: Construct a multimodal false information detection model, which includes a global feature extractor LSTM and a local feature extractor BERT for text, and a global feature extractor CNN and a local feature extractor Swin Transformer for images, which are used to extract the global feature V of the text respectively. g and text local features V l , image global feature U g and image local features U l ;

[0010] Step 3: Use the improved cosine distance to perform local semantic consistency detection on the extracted local features of the text and the local features of the image, and use the Euclidean distance to perform global semantic consistency detection on the global features of the text and the global features of the image. Combine the local semantic consistency and global semantic consistency detection results and generate a joint semantic consistency result S through the dynamic fusion coefficient;

[0011] Step 4: By assigning different weights, the text features and image features are effectively fused to generate the fused features F representing multimodal information;

[0012] Step 5: Concatenate and fuse the joint semantic consistency result S with the multimodal fusion feature F, and input the fusion result G into the two-layer perceptron to obtain the feature representation H2 after the two-layer perceptron mapping processing. Then input the feature representation H2 into the classification layer to obtain the false information discrimination result p c , p c is the probability that the predicted information is false, using p c Construct a loss function and use the training dataset in step 1 to train the false information detection model.

[0013] Preferably, in the step 2, the extracting of global features of the text specifically includes: traversing the text forward and backward through the global feature extractor LSTM to extract the global features V g The global features of the text include the dependency relationship between words in the text; the extraction of the global features of the image specifically includes: using a global feature extractor CNN to obtain the global features U of the image through convolution, pooling, and full connection layer operations g The extraction of local features of text specifically includes: encoding text information word by word through the local feature extractor BERT, extracting the representation of each word in the context, capturing the local information of each word in the context through BERT's own attention mechanism, and obtaining the local feature V of the text. l , where n is the number of words obtained after the word segmentation process in step 1. The phrase-level information of each word is obtained by convolution kernels of different sizes. t represents the word obtained after the word segmentation process in step 1, and v represents the local features of the word after word segmentation extracted after BERT processing. The specific formula is as follows:

[0014] V l ={v1…v n}=BERT({t1…t n})

[0015] The method of extracting local features of an image specifically includes: dividing the input image into local image blocks of fixed size by using a local feature extractor Swin Transformer, applying the sliding window mechanism of Swin Transformer itself in the local image blocks, extracting the features u of the local image blocks one by one, and combining the features extracted from each local image block to obtain the local features U of the image. l , captures the fine-grained information of the image, m represents the number of local image blocks, SWIN-T represents the Swin Transformer model, and e represents the local image block. The specific formula is as follows:

[0016] U l ={u1…u m}=SWIN-T({e1…e m}).

[0017] Preferably, the step three specifically includes:

[0018] For each local image block feature u i , the local features v of each word extracted in step 2 j and the feature u of the local image block i The improved cosine theorem is used to calculate the local semantic consistency score as follows:

[0019]

[0020] Take the maximum value of the local semantic consistency score as the semantic consistency result S of each local image block i , specifically expressed as:

[0021]

[0022] The local text image semantic consistency result S is obtained according to the following formula l (U l ,V l ):

[0023]

[0024] The text feature vector V is calculated as follows g and image feature vector U g The Euclidean distance in space is used to measure the consistency between modalities, and the weight matrix W is combined to adjust the consistency measurement result to obtain the global consistency information S g (V g ,U g ), and use the ReLU activation function to ensure that the consistency result value is non-negative:

[0025]

[0026] The joint semantic consistency result S is calculated according to the following formula. The dynamic fusion coefficient λ is used to fuse the global semantic consistency result and the local semantic consistency result. λ is a hyperparameter that adjusts the importance of the module in the overall model according to its size and training amount to maximize the overall model performance.

[0027] S=λ*S l (V l ,U l )+(1-λ)*S g (V g ,U g ).

[0028] Preferably, the step 4 specifically includes: extracting the global image feature U from the step 2g and text global feature A g Perform linear transformation to obtain U g ′ and V g ′, achieving the purpose of mapping image and text modalities to the same dimension;

[0029] The image modality and text modality are calculated through the self-attention mechanism to obtain the corresponding image and text attention weight matrices; according to the following formula, the weight matrix is ​​used to perform weighted calculation on the corresponding modality, and the image features and text features obtained by weighted calculation are fused: F = concat(A U′ U g ′,A V′ V g ′)

[0030] F=concat(A U′ U g ′,A V′ V g ′)

[0031] F represents the fused features, A U′ and A V′ Represent the image and text weight matrices after attention mechanism processing respectively.

[0032] Preferably, the step five specifically includes: fusing the joint semantic consistency result S obtained in step three with the multimodal fusion feature F in step four. The specific operation for information discrimination is as follows: splicing and fusing the joint semantic consistency result S with the multimodal fusion feature F, and inputting the fused result G into a two-layer perceptron to obtain a feature representation H2 after mapping processing by the two-layer perceptron, and then inputting the feature representation H2 into the classification layer for classification to obtain a false information discrimination result p c , p c is the probability that the predicted information is false;

[0033] G=concat(F,S)

[0034] H2=ReLU(W2·ReLU(W1·G+b1)+b2)

[0035] P c =σ(W o H2+b o )

[0036] W1,W2,,W o , represents 3 weight matrices, b1, b2, b o Indicates 3 bias values;

[0037] Define the cross-entropy loss function and the semantic consistency loss function, and use these two loss functions to construct a comprehensive cross-entropy loss function for training a multimodal false information detection model. The specific steps include the following:

[0038] Cross entropy loss function L cross_entropy The calculation formula is as follows:

[0039] L cross_entropy =-E[y·log p c +(1-y)·log(1-p c )]

[0040] L cross_entropy is the cross entropy loss function, which represents the difference between the prediction and the true value, E represents the expected value, which means that the average loss is calculated for the data in the entire training dataset, and y is the true label.

[0041] Considering the semantic consistency factor, construct the consistency loss L consistency :

[0042] L consistency =||FS|| 2

[0043] Combining the above two parts to obtain the final loss function L;

[0044] L=α·L cross_entropy +β·L consistency

[0045] Where L represents the final loss function, α + β = 1; using the training set described in step 1, the multimodal false information detection model is trained with the goal of minimizing the loss function L.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] In response to the problem that existing methods have difficulty in simultaneously processing fine-grained and coarse-grained multimodal information, this paper proposes to adopt local and global feature extraction methods to collect multimodal data at different granularities. To address the problem of cross-modal semantic association modeling, a global-local semantic consistency joint modeling method is adopted to fully capture the complex associations between image modality and text modality. By combining the deep fusion results of multimodal features and global and local joint consistency analysis, the present invention significantly improves the accuracy of multimodal false information detection. It also improves the robustness and generalization ability of the model in complex false news scenarios.

[0048] In response to the limitations of existing technologies in simultaneously processing fine-grained and coarse-grained multimodal information, the present invention proposes a local and global feature extraction method that can effectively collect multimodal data from different granularity levels. This method realizes the comprehensive processing of multimodal information by extracting fine-grained and coarse-grained features of multimodal data respectively. In response to the problem of cross-modal semantic association modeling, the present invention provides a global-local semantic consistency joint modeling method that can effectively capture the complex semantic association relationship between image modality and text modality, thereby improving the semantic understanding ability of cross-modal data. By combining the deep fusion results of multimodal features with global-local semantic consistency analysis, the present invention significantly improves the accuracy of multimodal false information detection, enhances the robustness and generalization ability of the model in complex false news scenarios, and is suitable for the field of multimodal data processing and false information detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 This is a flow chart of a multimodal false information detection method based on semantic consistency of the present invention;

[0051] Figure 2 This is a system architecture diagram of a multimodal false information detection method based on semantic consistency of the present invention;

[0052] Figure 3 This is a semantic consistency detection model diagram of a multimodal false information detection method based on semantic consistency in the present invention. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0054] Before introducing the technical solution of the present invention, the professional terms involved are first explained or illustrated: SwinTransformer: It is a vision-based Transformer architecture that processes local features through a layered window mechanism, divides the image into small blocks, and uses self-attention calculation to effectively capture image detail information.

[0055] BERT (Bidirectional Encoder Representations from Transformers): is a pre-trained language model that can effectively understand the meaning of words and sentences in text by taking context into account through a bidirectional Transformer architecture.

[0056] Self-Attention: It is a mechanism that calculates the similarity between each element in a sequence and other elements. It re-represents each element through weighted summation, allowing the model to focus on important information related to the current element in the sequence and capture the dependencies between elements.

[0057] Reference Figure 1 and Figure 2 A multimodal false information detection method based on semantic consistency includes four modules: feature extraction module, semantic consistency detection module, feature fusion module, and decision module. The functional modules are as follows:

[0058] (1) Feature extraction module: used to extract global and local features of text and image data, including two steps: data preprocessing and feature extraction. First, obtain news multimodal data from social networks and construct a training data set. Each data in the training data set includes news multimodal data and a label indicating whether it is false information. The news multimodal data includes text data and image data; perform word segmentation on the text data; perform word segmentation on the text data, break down the sentences into word units, and ensure that each text fragment can be converted into a vector representation that can be processed by the model. In addition, in order to meet the input requirements of the model, the length of the text sequence is controlled, the maximum sequence length is retained, the excess part is truncated, and padding is performed when the length is insufficient. For image data, the image is first converted to a uniform size to adapt to the dimension of the model input layer; secondly, a multimodal false information detection model is constructed for the preprocessed data. The multimodal false information detection model includes a global feature extractor LSTM and a local feature extractor BERT for text, and a global feature extractor CNN and a local feature extractor Swin Transformer for images, which are used to extract the global feature V of the text respectively. g and text local features V l , image global feature U g and image local features U l ;

[0059] ① Extract global features of text: By traversing the preprocessed text data in both forward and reverse directions using bidirectional LSTM, we can simultaneously obtain the contextual information of the previous and following texts, including the dependency relationship between all words in the sentence, to obtain the global features V of the text.g ;

[0060] ②For global feature extraction of images: CNN is used to obtain the global feature U of the image through convolution, pooling, and full connection layer operations. g ;

[0061] ③ Extracting local information from text: BERT is used to encode the input text word by word, extract the representation of each word in the context, and use BERT’s own attention mechanism to capture the local information of each word in the context, and obtain the local feature V of the text. l , where n is the length of the text, phrase-level information of each word is obtained through convolution kernels of different sizes, t is the text word before processing, and v is the text word after BERT processing. The specific formula is as follows:

[0062] V l ={v1…v n}=BERT({t1…t n})

[0063] ④ For local image information extraction: The preprocessed image is divided into fixed-size image blocks through Swin Transformer. The self-attention mechanism of the sliding window is applied to these local image blocks to extract the features of the local area block by block to capture the fine-grained information of the image. m represents the number of local image blocks. The formula is as follows:

[0064] U l ={u1…u m}=SWIN-T({e1…e m}).

[0065] (2) Semantic consistency detection module: used to detect semantic consistency between modalities, such as Figure 3 As shown. The local features v extracted by the feature extraction module (1) are j and the local image feature u i The improved cosine theorem is used for local semantic consistency detection. The global features of the text and the global features of the image are tested for global semantic consistency using the improved Euclidean distance. The local and global semantic consistency detection results are combined to generate the final semantic consistency result through the dynamic fusion coefficient.

[0066] ①: Local semantic consistency detection: Further, the extracted text local features v j and the local image feature u i The improved cosine theorem is used to detect semantic consistency between text words and image blocks. The specific calculation formula is as follows:

[0067]

[0068] The above-mentioned further solution has the beneficial effect of modifying the cosine theorem to ensure that the semantic consistency results fall within the range [0, 1]. This removes negative values ​​to avoid misjudgment of information due to excessively high or low similarity scores, and simplifies the subsequent local semantic consistency detection calculations. By calculating the cosine distance between text words and image blocks, the semantic consistency between the two can be better measured.

[0069] After obtaining the cosine distance results between the text words and all image blocks, the maximum value is taken to represent the semantic consistency of each area. The specific expression is:

[0070]

[0071] The overall local text image semantic consistency result is expressed as:

[0072]

[0073] The beneficial effect of the above scheme is that by taking the maximum value of the similarity between each text word and image block to represent the semantic consistency result between the two, the area with the strongest semantic association is highlighted to avoid the influence of the inconsistent semantic degree of all image blocks, and the degree of alignment between the text and image semantics is effectively represented. Finally, by averaging the semantic consistency results of the entire area, the contribution of each local feature is taken into account, while avoiding the influence of the overall local semantic consistency detection result due to the occasional high or low semantic consistency score of a single text and image block.

[0074] ② Global semantic consistency detection: Further, by calculating the text feature vector V g and image feature vector U g The distance between them is calculated and the weight matrix W is combined to adjust the consistency measurement result to obtain consistency information. In addition, the normalization factor ||W||·(||V g ||2+||U g ||2) is adjusted to ensure rationality under the difference in feature size, and the ReLU activation function is used to ensure that the consistency result value is non-negative, quantifying the global consistency between text and image, reflecting their semantic consistency. The specific formula is expressed as:

[0075]

[0076] The beneficial result of the above further scheme is that by drawing on the Euclidean distance idea to calculate the difference between text and image features, combined with the weight W adjustment, the ReLU activation function is introduced to correct the result and eliminate negative values ​​to obtain the semantic consistency detection result, and the semantic consistency of the two modalities in multiple dimensions of feature scale and direction is measured from a global perspective.

[0077] ③: Joint Semantic Consistency Detection: To achieve joint semantic consistency, a dynamic fusion method is proposed to flexibly allocate weights between global and local semantic consistency, improving matching accuracy in complex scenarios. λ is a hyperparameter that adjusts the importance of a module in the overall model based on its size and training volume, maximizing overall model performance. In this embodiment, the initial value of λ is set to 0.5. When the modality is relatively abstract, it can be set to 0.3-0.4, and when the modality is more specific, it can be set to 0.6-0.7.

[0078] The specific formula is as follows:

[0079] S=λ*S l (V l ,U l )+(1-λ)*S g (V g ,U g )

[0080] The beneficial effect of this approach is that when the modality is relatively abstract, the representation of the local region may not be sufficient to effectively capture the degree of matching. Therefore, it is necessary to reduce the weight of local semantic consistency and increase the influence of global semantic consistency. When the modality is relatively specific, the weight of global semantic consistency is reduced and the weight of local semantic consistency is increased to capture feature details. This method dynamically and flexibly adjusts the results of local and global semantic consistency to avoid the influence of overly abstract or specific modalities on semantic consistency results. It can effectively capture the relationship between text and semantics and achieve accurate joint semantic consistency results.

[0081] (3) Feature fusion module, which combines the global feature U of the image processed by (1) g and text global feature V g The input is fed into the feature fusion module to effectively fuse the text features with the image features by assigning different weights, generating fusion features representing multimodal information as follows:

[0082] First, the image global feature U g and text global feature V g Perform linear transformation to map the image and text modalities to the same dimension and obtain U g ′ and V g ′, the image modality and text modality are calculated through the self-attention mechanism and the corresponding attention weight matrix is ​​obtained. The weight matrix is ​​used to perform weighted calculation on the corresponding modality, and the image features and text features obtained by weighted calculation are fused, which can be expressed as:

[0083] F=concat(A U′ U g ′,AV′ V g ′)

[0084] F represents the fused features, A U′ and A V′ Represent the image and text weight matrices after attention mechanism processing respectively.

[0085] (4) Decision module: Combine the results of multimodal fusion and semantic consistency to judge the authenticity of information and establish a loss function to optimize the model.

[0086] First, the joint semantic consistency detection result S obtained by (2) and the feature fusion result F obtained by (3) are spliced ​​and fused, and the fused result is input into the perceptron. The present invention adopts a two-layer perceptron. The feature representation after perceptron mapping is input into the classification layer to obtain the probability p that the information is false. c , the embodiment of the present invention sets the threshold value to 0.5 when p c When it is greater than 0.5, the information is false, otherwise it is true.

[0087] G=concat(F,S)

[0088] H2=ReLU(W2·ReLU(W1·G+b1)+b2)

[0089] P c =σ(W o H2+b o )

[0090] G represents the feature after the fusion of F and S, H2 represents the feature representation after the two-layer perceptron processing, W1, W2,, W o , represents the weight matrix, b1, b2, b o Indicates the offset value.

[0091] The beneficial effect of this approach lies in leveraging the complementary strengths of both. Multimodal features provide rich signals from diverse information sources, helping to comprehensively capture the multidimensional characteristics of information, thereby improving the model's ability to identify false information. Semantic consistency information, on the other hand, measures the consistency between different modalities, helping the model correct potential contradictions or inconsistencies between them and enhancing its overall understanding of information authenticity. Combining these two approaches not only improves the accuracy and robustness of false information detection, but also effectively addresses the complexity and ambiguity inherent in false information, resulting in more accurate and stable classification results.

[0092] The cross entropy loss function and semantic consistency loss function are defined and a comprehensive cross entropy loss function is constructed using these two loss functions to optimize the false information detection model. The specific operations are as follows;

[0093] The cross entropy loss function is specifically expressed as:

[0094] L cross_entropy =-E[y·log p c +(1-y)·log(1-p c )]

[0095] L cross_entropy The cross entropy loss function represents the difference between the prediction and the true value, E represents the expected value, which means that the average loss is calculated for the data in the entire training dataset, y is the true label, and p c is the probability that the predicted information is false;

[0096] To further consider the semantic consistency between images and texts, a consistency loss formula is defined to calculate the distance between the modality fusion result and the semantic consistency result as follows:

[0097] L consistency =||FS|| 2

[0098] F represents the fused features, S represents the joint semantic consistency detection results, and L consistency Represents the consistency loss function.

[0099] The beneficial effect of the above scheme is that by calculating the distance between the multimodal fusion feature F and the semantic consistency detection result S, a consistency loss function is established. On the one hand, it further captures the degree of semantic matching between text and image. When F and S differ greatly, it indicates that the consistency information between the two modalities is not fully captured during the fusion process. On the other hand, it provides additional constraints for the model to prevent the model from relying too much on F or S to prevent overfitting.

[0100] Finally, combining the above two parts can get the final loss function;

[0101] L=α·L cross_entropy +β·L consistency

[0102] Where L represents the final loss function and α + β = 1; by minimizing the loss function, the model will learn how to make better predictions given the input.

[0103] In response to the limitations of existing technologies in simultaneously processing fine-grained and coarse-grained multimodal information, the present invention proposes a local and global feature extraction method that can effectively collect multimodal data from different granularity levels. This method realizes the comprehensive processing of multimodal information by extracting fine-grained and coarse-grained features of multimodal data respectively. In response to the problem of cross-modal semantic association modeling, the present invention provides a global-local semantic consistency joint modeling method that can effectively capture the complex semantic association relationship between image modality and text modality, thereby improving the semantic understanding ability of cross-modal data. By combining the deep fusion results of multimodal features with global-local semantic consistency analysis, the present invention significantly improves the accuracy of multimodal false information detection, enhances the robustness and generalization ability of the model in complex false news scenarios, and is suitable for the field of multimodal data processing and false information detection.

[0104] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It is apparent to those skilled in the art that various changes, modifications, substitutions, and variations to these embodiments may be made without departing from the principles and spirit of the present invention, and these changes and modifications still fall within the scope of protection of the present invention.

Claims

1. A multimodal false information detection method based on semantic consistency, characterized by: The following steps are involved: Step 1: Obtain multimodal news data from social networks and construct a training dataset. Each data in the training dataset includes multimodal news data and a label indicating whether it is false information. The multimodal news data includes text data and image data; perform word segmentation on the text data. Step 2: Construct a multimodal false information detection model, which includes a global feature extractor LSTM and a local feature extractor BERT for text, and a global feature extractor CNN and a local feature extractor Swin Transformer for images, which are used to extract the global feature V of the text respectively. g and text local features V l , image global feature U g and image local features U l ; Step 3: Use the improved cosine distance to perform local semantic consistency detection on the extracted local features of the text and the local features of the image, and use the Euclidean distance to perform global semantic consistency detection on the global features of the text and the global features of the image. Combine the local semantic consistency and global semantic consistency detection results and generate a joint semantic consistency result S through the dynamic fusion coefficient; Step 4: By assigning different weights, the text features and image features are effectively fused to generate the fused features F representing multimodal information; Step 5: Concatenate and fuse the joint semantic consistency result S with the multimodal fusion feature F, and input the fusion result G into the two-layer perceptron to obtain the feature representation H2 after the two-layer perceptron mapping processing. Then input the feature representation H2 into the classification layer to obtain the false information discrimination result p c , p c is the probability that the predicted information is false, using p c Construct a loss function and use the training dataset in step 1 to train the false information detection model.

2. The multimodal false information detection method based on semantic consistency according to claim 1, characterized in that: In the step 2: The extraction of global features of the text specifically includes: traversing the text forward and backward through the global feature extractor LSTM to extract the global features V g ,The global features of the text include the dependency relations between words in the text; The method of extracting the global features of the image specifically includes: using a global feature extractor CNN to obtain the global features U of the image through convolution, pooling, and full connection layer operations g ; The method of extracting local features of text specifically includes: encoding text information word by word through the local feature extractor BERT, extracting the representation of each word in the context, and capturing the local information of each word in the context through BERT's own attention mechanism to obtain the local features of the text V l , where n is the number of words obtained after the word segmentation process in step 1. The phrase-level information of each word is obtained by convolution kernels of different sizes. t represents the word obtained after the word segmentation process in step 1, and v represents the local features of the word after word segmentation extracted after BERT processing. The specific formula is as follows: V l ={v1…v n }=BERT({t1…t n }) The method of extracting local features of an image specifically includes: dividing the input image into local image blocks of fixed size by using a local feature extractor Swin Transformer, applying the sliding window mechanism of Swin Transformer itself in the local image blocks, extracting the features u of the local image blocks one by one, and combining the features extracted from each local image block to obtain the local features U of the image. l , m represents the number of local image blocks, SWIN-T represents the Swin Transformer model, and e represents the local image block. The specific formula is as follows: HE l {u1...u m }=SWIN-T({e1…e m })。 3. The multimodal false information detection method based on semantic consistency according to claim 2, characterized in that: The step three specifically includes: For each local image block feature u i , the local features v of each word extracted in step 2 j and the feature u of the local image block i The improved cosine theorem is used to calculate the local semantic consistency score as follows: Take the maximum value of the local semantic consistency score as the semantic consistency result S of each local image block i , specifically expressed as: The local text image semantic consistency result S is obtained according to the following formula l (U l ,V l ): Calculate the text feature vector V according to the following formula g and image feature vector U g The Euclidean distance in space is used to measure the consistency between modalities, and the weight matrix W is combined to adjust the consistency measurement result to obtain the global consistency information S g (V g ,U g ), and use the ReLU activation function to ensure that the consistency result value is non-negative: The joint semantic consistency result S is calculated according to the following formula. The dynamic fusion coefficient λ is used to fuse the global semantic consistency result and the local semantic consistency result. S=λ*S l (V l ,U l )+(1-λ)*S g (V g ,U g )。 4. The multimodal false information detection method based on semantic consistency according to claim 3 is characterized by: The step 4 specifically includes: The global image feature U extracted in step 2 is g and text global feature V g Perform linear transformation to obtain U′ g and V′ g , achieving the goal of mapping image and text modalities to the same dimension; The image modality and text modality are calculated through the self-attention mechanism to obtain the corresponding image and text attention weight matrices. According to the following formula, the weight matrix is ​​used to perform weighted calculation on the corresponding modality, and the image features and text features obtained by weighted calculation are fused: F=comcat(A U′ U′ g ,A V′ V′ g ) F represents the fused features, A U′ and A V′ Represent the image and text attention weight matrices after attention mechanism processing respectively.

5. The multimodal false information detection method based on semantic consistency according to claim 4 is characterized in that: The step five specifically includes: The joint semantic consistency result S obtained in step 3 is fused with the multimodal fusion feature F in step 4. The specific operation for information discrimination is as follows: the joint semantic consistency result S is spliced ​​and fused with the multimodal fusion feature F, and the fusion result G is input into the two-layer perceptron to obtain the feature representation H2 after the mapping processing of the two-layer perceptron. The feature representation H2 is then input into the classification layer to obtain the false information discrimination result p. c , p c is the probability that the predicted information is false; G=concat(F,S) H2=ReLU(W2·ReLU(W1·G+b1)+b2) P c =σ(W o ·H2+b o ) W1,W2,W o Represents 3 weight matrices, b1, b2, b o Indicates 3 bias values; Define the cross-entropy loss function and the semantic consistency loss function, and use these two loss functions to construct a comprehensive cross-entropy loss function for training a multimodal false information detection model. The specific steps include the following: Cross entropy loss function L cross_entropy The calculation formula is as follows: L cross_entropy =-E[y·logp c +(1-y)·log(1-p c )] L cross_entropy is the cross entropy loss function, which represents the difference between the prediction and the true value, E represents the expected value, which means that the average loss is calculated for the data in the entire training dataset, and y is the true label. Construct consistency loss L consistency : L consistency =||F-S|| 2 Combining the above two parts to obtain the final loss function L; L=α·L cross_entropy +β·L consistency Where L represents the final loss function, α + β = 1; using the training set described in step 1, the multimodal false information detection model is trained with the goal of minimizing the loss function L.

Images