Multi-modal false information detection method based on semantic consistency

Abstract

The invention relates to a multi-modal false information detection method based on semantic consistency, and aims to solve the problem of semantic interaction between modals in multi-modal false information detection. The method specifically comprises the following steps: firstly, acquiring multi-modal data of news in a social network, and preprocessing a text and an image; then respectively constructing global and local feature extractors of the text and the image; local semantic consistency detection is carried out through the extracted text local features and the image local features, global semantic consistency detection is carried out on the obtained global features, the two features are combined to generate a joint semantic consistency result, and the attention mechanism text and image features are adopted to carry out multi-modal fusion to obtain a joint semantic consistency result; the false information is distinguished from multiple dimensions and a loss function is established to optimize the model by comprehensively analyzing the multi-modal features after the text features and the image features are fused and the united semantic consistency result, so that the detection of the false information is realized. According to the method, the accuracy of multi-modal false information detection can be effectively improved, and the multi-modal false information can be automatically detected.

Description

Technical Field

[0001] The invention relates to a multimodal false information detection method based on semantic consistency, and belongs to the technical field of false information detection. Background Art

[0002] In recent years, the rapid development of social networks has made the way users express their emotions and opinions more diverse, making it increasingly difficult to discern the authenticity of information. Fake news has evolved from the traditional pure text form to a multimodal presentation method, adding multiple elements such as visual images. This multimodal false information is not only more attractive, but also spreads significantly faster, bringing far-reaching harm to the network ecology and seriously affecting social stability and people's daily lives. Therefore, how to automatically detect multimodal false news has become an important issue that needs to be solved in the field of social network analysis, and has important research value and practical application significance.

[0003] There are complex associations between multimodal data, and the feature relationships within and between modalities are also extremely complex. Traditional feature interaction methods have difficulty capturing such complex high-dimensional interactive relationships. For example, there is often a fine-grained local correspondence between information text and accompanying pictures, and this relationship has a certain correlation; at the same time, the overall content of the text and image complement each other to express the content of the text and image in more detail.

[0004] Most existing multimodal fake news detection methods only focus on information interaction within the modality, or the fusion of global information, and often ignore the relationship between local information and global information. Although these methods show a certain simplicity and effectiveness in dealing with a large number of multimodal fake news cases, they lack in-depth consideration of the interaction of information at different granularities and are difficult to fully integrate fine-grained and coarse-grained features. In actual scenarios, fake news often causes contradictions between local information and overall semantics by making subtle changes to real image and text information. Therefore, traditional cross-modal semantic association learning methods, which are global or local oriented, are difficult to effectively model the semantic interaction between image regions and text words, which makes the detection model face challenges in reasoning and judgment between heterogeneous modalities.

[0005] In order to effectively utilize multimodal data, it is crucial to deeply explore the semantic associations between different modalities for the detection of multimodal false information. This requires researchers to explore more complex models that can simultaneously consider the interaction of local and global information, and improve the accuracy and robustness of false news detection through multi-level semantic learning. With the help of advanced technologies such as deep learning, building a system that can capture the deep relationship between multimodal features will provide a powerful tool for combating false news. Summary of the invention

[0006] In order to overcome the problem of semantic interaction between modalities in the existing multimodal false information detection, the present invention provides a multimodal false information detection method based on semantic consistency. Specifically comprising the following steps: first, obtain the multimodal data of news in the social network, and preprocess the text and image. Then construct the global and local feature extractors of the text and image respectively. Perform local semantic consistency detection on the extracted local features of the text and the local features of the image, perform global semantic consistency detection on the acquired global features, and generate a joint semantic consistency result. Then use the attention mechanism to assign weights and perform multimodal fusion of text and image features. Finally, combine the fusion result with the semantic consistency result to establish a loss function, thereby realizing the detection of false information. The present invention can effectively improve the accuracy of multimodal false information detection and improve the automatic detection of multimodal false information.

[0007] A multimodal false information detection method based on semantic consistency comprises the following steps:

[0008] Step 1: Obtain news multimodal data from social networks and construct a training data set, where each data in the training data set includes news multimodal data and a label of whether it is false information, and the news multimodal data includes text data and image data; perform word segmentation on the text data;

[0009] Step 2: Construct a multimodal false information detection model, which includes a global feature extractor LSTM and a local feature extractor BERT for text, and a global feature extractor CNN and a local feature extractor Swin Transformer for images, which are used to extract text global features V g and text local features V l , image global feature U g and the local image feature U l ;

[0010] Step 3: Use the improved cosine distance to perform local semantic consistency detection on the extracted text local features and image local features, and use the Euclidean distance to perform global semantic consistency detection on the text global features and image global features. Combine the local semantic consistency and global semantic consistency detection results, and generate a joint semantic consistency result S through a dynamic fusion coefficient;

[0011] Step 4: By assigning different weights, the text features and image features are effectively fused to generate fused features F representing multimodal information;

[0012] Step 5: Concatenate and fuse the joint semantic consistency result S with the multimodal fusion feature F, and input the fused result G into the two-layer perceptron to obtain the feature representation H2 after the two-layer perceptron mapping processing, and then input the feature representation H2 into the classification layer to obtain the false information discrimination result p c , p c is the probability that the predicted information is false, using p c Construct a loss function and use the training data set in step 1 to train the false information detection model.

[0013] Preferably, in step 2: extracting the global features of the text specifically includes: traversing the text forward and backward through the global feature extractor LSTM to extract the global features V of the text g , the global features of the text include the dependency relationship between words in the text; the extraction of the global features of the image specifically includes: using a global feature extractor CNN to obtain the global features U of the image through convolution, pooling, and full connection layer operations g ; The extracting of local features of text specifically includes: encoding text information word by word through the local feature extractor BERT, extracting the representation of each word in the context, capturing the local information of each word in the context through BERT's own attention mechanism, and obtaining the local feature V of the text l , where n is the number of words obtained after the word segmentation processing in step 1, the phrase-level information of each word is obtained through convolution kernels of different sizes, t represents the word obtained after the word segmentation processing in step 1, and v represents the local features of the word after word segmentation extracted after BERT processing. The specific formula is as follows:

[0014] V l = {v1…v n}=BERT({t1…t n})

[0015] The method of extracting local features of an image specifically includes: dividing the input image into local image blocks of fixed size by using a local feature extractor Swin Transformer, applying the sliding window mechanism of Swin Transformer itself in the local image blocks, extracting features u of the local image blocks one by one, and combining the features extracted from each local image block to obtain the local feature U of the image. l , captures the fine-grained information of the image, m represents the number of local image blocks, SWIN-T represents the Swin Transformer model, and e represents the local image block. The specific formula is as follows:

[0016] U l = {u1…u m}=SWIN-T({e1…e m}).

[0017] Preferably, the step three specifically includes:

[0018] For each local image block feature u i , the local features v of each word extracted in step 2 j and the feature u of the local image block i The local semantic consistency score is calculated using the improved cosine theorem as follows:

[0019]

[0020] Take the maximum value of the local semantic consistency score as the semantic consistency result S of each local image block i , specifically expressed as:

[0021]

[0022] The local text image semantic consistency result S is obtained according to the following formula l (U l ,V l ):

[0023]

[0024] The text feature vector V is calculated as follows g and image feature vector U g The Euclidean distance in space is used to measure the consistency between modalities, and the weight matrix W is combined to adjust the consistency measurement result to obtain the global consistency information S g (V g ,U g ), and use the Relu activation function to ensure that the consistency result value is non-negative:

[0025]

[0026] The joint semantic consistency result S is calculated according to the following formula. The dynamic fusion coefficient λ is used to fuse the global semantic consistency result and the local semantic consistency result. λ is a hyperparameter that adjusts the importance of the module in the overall model according to its size and training amount, so as to maximize the performance of the overall model.

[0027] S=λ*S l (V l ,U l )+(1-λ)*S g (V g ,U g ).

[0028] Preferably, the step 4 specifically includes: extracting the image global feature U from step 2g and text global feature A g Perform linear transformation to obtain U g ′ and V g ′, achieving the purpose of mapping image and text modalities to the same dimension;

[0029] The image modality and text modality are calculated through the self-attention mechanism to obtain the corresponding image and text attention weight matrices; according to the following formula, the weight matrix is ​​used to perform weighted calculation on the corresponding modality, and the image features and text features obtained by weighted calculation are fused: F = concat(A U′ U g ′,A V′ V g ′)

[0030] F = concat(A U′ U g ′,A V′ V g ′)

[0031] F represents the fused features, A U′ and A V′ Respectively represent the image and text weight matrices after being processed by the attention mechanism.

[0032] Preferably, the step five specifically includes: fusing the joint semantic consistency result S obtained in step three with the multimodal fusion feature F in step four. The specific operation for information discrimination is as follows: splicing and fusing the joint semantic consistency result S with the multimodal fusion feature F, and inputting the fused result G into a two-layer perceptron to obtain a feature representation H2 after mapping processing by the two-layer perceptron, and then inputting the feature representation H2 into the classification layer for classification to obtain a false information discrimination result p c , p c is the probability that the predicted information is false;

[0033] G = concat(F,S)

[0034] H2=ReLU(W2·ReLU(W1·G+b1)+b2)

[0035] P c =σ(W o ·H2+b o )

[0036] W1,W2,,W o , represents three weight matrices, b1, b2, b o Indicates 3 bias values;

[0037] Define the cross entropy loss function and the semantic consistency loss function, and use these two loss functions to construct a comprehensive cross entropy loss function for training a multimodal false information detection model, which specifically includes the following steps;

[0038] Cross entropy loss function L cross_entropy The calculation formula is as follows:

[0039] L cross_entropy = -E[y·log p c +(1-y)·log(1-p c )]

[0040] L cross_entropy is the cross entropy loss function, which represents the difference between the prediction and the true value. E represents the expected value, which means that the average loss is calculated for the data in the entire training dataset. y is the true label.

[0041] Considering the semantic consistency factor, construct the consistency loss L consistency :

[0042] L consistency =||FS|| 2

[0043] Combining the above two parts, we get the final loss function L;

[0044] L=α·L cross_entropy +β·L consistency

[0045] Where L represents the final loss function, α+β=1; using the training set described in step 1, the multimodal false information detection model is trained with the goal of minimizing the loss function L.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] In view of the problem that existing methods are difficult to simultaneously process fine-grained and coarse-grained multimodal information, this paper proposes to adopt local and global feature extraction methods to collect multimodal data from different granularities. In view of the problem of cross-modal semantic association modeling, a global-local semantic consistency joint modeling method is adopted to fully capture the complex association between image modality and text modality. The present invention significantly improves the accuracy of multimodal false information detection by combining the deep fusion results of multimodal features and global and local joint consistency analysis. The robustness and generalization ability of the model in complex false news scenarios are improved.

[0048] In view of the limitations of the prior art in simultaneously processing fine-grained and coarse-grained multimodal information, the present invention proposes a local and global feature extraction method that can effectively collect multimodal data from different granularity levels. This method realizes the comprehensive processing of multimodal information by extracting fine-grained and coarse-grained features of multimodal data respectively. In view of the problem of cross-modal semantic association modeling, the present invention provides a global-local semantic consistency joint modeling method that can effectively capture the complex semantic association relationship between image modality and text modality, thereby improving the semantic understanding ability of cross-modal data. By combining the deep fusion results of multimodal features with global-local semantic consistency analysis, the present invention significantly improves the accuracy of multimodal false information detection, enhances the robustness and generalization ability of the model in complex false news scenarios, and is suitable for the fields of multimodal data processing and false information detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 A flowchart of a multimodal false information detection method based on semantic consistency of the present invention;

[0051] Figure 2 This is a system architecture diagram of a multimodal false information detection method based on semantic consistency of the present invention;

[0052] Figure 3 This is a semantic consistency detection model diagram of a multimodal false information detection method based on semantic consistency of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] Before introducing the technical solution of the present invention, the professional terms involved are first explained or illustrated: SwinTransformer: It is a vision-based Transformer architecture, which processes local features through a layered window mechanism, divides the image into small blocks, and uses self-attention calculation to effectively capture image detail information.

[0055] BERT (Bidirectional Encoder Representations from Transformers): is a pre-trained language model that can effectively understand the meaning of words and sentences in text by taking into account contextual information through a bidirectional Transformer architecture.

[0056] Self-Attention Mechanism: It is a mechanism that calculates the similarity between each element in a sequence and other elements, and re-represents each element by weighted summation, so that the model can focus on the important information related to the current element in the sequence and capture the dependencies between elements.

[0057] Reference Figure 1 and Figure 2 A multimodal false information detection method based on semantic consistency includes four modules: feature extraction module, semantic consistency detection module, feature fusion module, and decision module. The functional modules are as follows:

[0058] (1) Feature extraction module: used to extract global and local features of text and image data, including two steps: data preprocessing and feature extraction. First, obtain news multimodal data from social networks and construct a training data set. Each data in the training data set includes news multimodal data and a label indicating whether it is false information. The news multimodal data includes text data and image data; perform word segmentation on the text data; perform word segmentation on the text data and break down sentences into word units to ensure that each text fragment can be converted into a vector representation that can be processed by the model. In addition, in order to meet the input requirements of the model, the length of the text sequence is controlled, the maximum sequence length is retained, the excess part is truncated, and padding is performed when the length is insufficient. For image data, first convert the image to a uniform size to adapt to the dimension of the model input layer; secondly, construct a multimodal false information detection model for the preprocessed data. The multimodal false information detection model includes a global feature extractor LSTM and a local feature extractor BERT for text, and a global feature extractor CNN and a local feature extractor Swin Transformer for images, which are used to extract text global features V respectively. g and text local features V l , image global feature U g and the local image feature U l ;

[0059] ① Extract global features of text: By traversing the preprocessed text data in the forward and reverse directions through the bidirectional LSTM, the context information of the previous and following texts can be obtained at the same time, including the dependency relationship between all words in the sentence, to obtain the global features V of the text.g ;

[0060] ②For global feature extraction of images: CNN is used to obtain the global feature U of the image through convolution, pooling, and full connection layer operations. g ;

[0061] ③ Extracting local information from text: Encode the input text information word by word through BERT, extract the representation of each word in the context, and capture the local information of each word in the context through BERT’s own attention mechanism to obtain the local feature V of the text. l , where n is the length of the text, the phrase-level information of each word is obtained through convolution kernels of different sizes, t is the text word before processing, and v is the text word after BERT processing. The specific formula is as follows:

[0062] V l = {v1…v n}=BERT({t1…t n})

[0063] ④ Extraction of local information of the image: The preprocessed image is divided into image blocks of fixed size through Swin Transformer. The self-attention mechanism of the sliding window is applied in these local image blocks to extract the features of the local area block by block to capture the fine-grained information of the image. The formula for m representing the number of local image blocks is as follows:

[0064] U l = {u1…u m}=SWIN-T({e1…e m}).

[0065] (2) Semantic consistency detection module: used to detect the semantic consistency between modalities, such as Figure 3 As shown in the figure, the local text feature v extracted by the feature extraction module (1) is j and the local image feature u i The improved cosine theorem is used to detect local semantic consistency. The global features of the text and the global features of the image are tested for global semantic consistency through the improved Euclidean distance. The local semantic consistency and global semantic consistency test results are combined to generate the final semantic consistency result through the dynamic fusion coefficient.

[0066] ①: Local semantic consistency detection: Further, the extracted text local features v j and the local image feature u i The improved cosine theorem is used to detect the semantic consistency between text words and image blocks. The specific calculation formula is as follows:

[0067]

[0068] The above further scheme has the beneficial effect of ensuring that the semantic consistency result range falls between [0,1] by modifying the cosine theorem, removing negative values ​​to avoid misjudgment of information due to too high or too low similarity scores, and simplifying the subsequent local semantic consistency detection calculation. The semantic consistency between the text word and the image block can be better measured by calculating the cosine distance between the two.

[0069] After obtaining the cosine distance results between the text word and all image blocks, the maximum value is taken to represent the semantic consistency of each area. The specific expression is:

[0070]

[0071] The overall local text image semantic consistency result is expressed as:

[0072]

[0073] The beneficial effect of the above scheme is that by taking the maximum value of the similarity between each text word and image block to represent the semantic consistency result between the two, the area with the strongest semantic association is highlighted to avoid the influence of inconsistent semantic levels of all image blocks, and the degree of alignment between text and image semantics is effectively represented. Finally, by averaging the semantic consistency results of the entire area, the contribution of each local feature is taken into account, while avoiding the influence of the overall local semantic consistency detection result due to the occasional high or low semantic consistency scores of a single text and image block.

[0074] ② Global semantic consistency detection: Further, by calculating the text feature vector V g and image feature vector U g The distance between them is calculated by combining the weight matrix W to adjust the consistency measurement result, so as to obtain the consistency information. In addition, the normalization factor ||W||·(||V g ||2+||U g ||2) is adjusted to ensure rationality under feature size differences, and the Relu activation function is used to ensure that the consistency result value is non-negative, quantifying the degree of global consistency between text and image, reflecting their semantic consistency. The specific formula is expressed as:

[0075]

[0076] The beneficial result of the above further scheme is that by borrowing the idea of ​​Euclidean distance to calculate the difference between text and image features, combined with the adjustment of weight W, the Relu activation function is introduced to correct the result and eliminate negative values ​​to obtain the semantic consistency detection result, and the semantic consistency of the feature scale and direction of the two modalities is measured from a global perspective.

[0077] ③: Joint semantic consistency detection: In order to obtain joint semantic consistency results, a dynamic fusion method is proposed to flexibly allocate weights between global semantic consistency and local semantic consistency to improve the matching accuracy in complex scenarios. λ is a hyperparameter that adjusts the importance of the module in the overall model according to its size and training amount to maximize the performance of the overall model. In the embodiment of the present invention, the initial value of λ is set to 0.5, which can be set to 0.3-0.4 when the modality is more abstract, and 0.6-0.7 when the modality is more specific.

[0078] The specific formula is as follows:

[0079] S=λ*S l (V l ,U l )+(1-λ)*S g (V g ,U g )

[0080] The beneficial effect of the above scheme is that when the modality is more abstract, the representation of the local area may not be sufficient to effectively capture the degree of matching. Therefore, it is necessary to reduce the weight of local semantic consistency and increase the influence of global semantic consistency. When the modality is more specific, reduce the weight of global semantic consistency and increase the weight of local semantic consistency to capture feature details. This method can effectively obtain the relationship between text and semantics and obtain accurate joint semantic consistency results by dynamically and flexibly adjusting the local semantic consistency and global semantic consistency results to avoid the influence of the modality being too abstract or specific on the semantic consistency results.

[0081] (3) Feature fusion module, which combines the global image features U processed by (1) g and text global feature V g The input is sent to the feature fusion module to effectively fuse the text features with the image features by assigning different weights, and generate fusion features representing multimodal information as follows:

[0082] First, the image global feature U g and text global feature V g Perform linear transformation to map the image and text modalities to the same dimension and obtain U g ′ and V g ′, the image modality and text modality are calculated through the self-attention mechanism and the corresponding attention weight matrix is ​​obtained. The weight matrix is ​​used to perform weighted calculation on the corresponding modality, and the image features and text features obtained by weighted calculation are fused, which can be expressed as:

[0083] F = concat(A U′ U g ′,AV′ V g ′)

[0084] F represents the fused features, A U′ and A V′ Respectively represent the image and text weight matrices after being processed by the attention mechanism.

[0085] (4) Decision-making module: Combine the results of multimodal fusion and semantic consistency to judge the authenticity of information and establish a loss function to optimize the model.

[0086] First, the joint semantic consistency detection result S obtained by (2) and the feature fusion result F obtained by (3) are concatenated and fused, and the fused result is input into the perceptron. The present invention adopts a two-layer perceptron. The feature representation after perceptron mapping is input into the classification layer to obtain the probability p of the information being false. c , the embodiment of the present invention sets the threshold value to 0.5 when p c When it is greater than 0.5, the information is false, otherwise it is true.

[0087] G = concat(F,S)

[0088] H2=ReLU(W2·ReLU(W1·G+b1)+b2)

[0089] P c =σ(W o ·H2+b o )

[0090] G represents the feature after the fusion of F and S, H2 represents the feature representation after being processed by two layers of perceptrons, W1, W2,, W o , represents the weight matrix, b1, b2, b o Indicates the offset value.

[0091] The beneficial effect of the above scheme is that it gives full play to the complementary advantages of the two. Multimodal features provide rich signals from different information sources, which helps to fully capture the multi-dimensional characteristics of information, thereby improving the model's ability to identify false information; while semantic consistency information helps the model correct potential contradictions or inconsistencies between modalities by measuring the consistency between different modalities, thereby enhancing the overall grasp of the authenticity of information. Combining the two can not only improve the accuracy and robustness of false information detection, but also effectively deal with the complexity and ambiguity that may exist in false information, thereby achieving more accurate and stable classification results.

[0092] The cross entropy loss function and the semantic consistency loss function are defined and a comprehensive cross entropy loss function is constructed using these two loss functions to optimize the false information detection model. The specific operations are as follows;

[0093] The cross entropy loss function is specifically expressed as:

[0094] L cross_entropy = -E[y·log p c +(1-y)·log(1-p c )]

[0095] L cross_entropy The cross entropy loss function represents the difference between the prediction and the true value, E represents the expected value, which means that the average loss is calculated for the data in the entire training dataset, y is the true label, and p c is the probability that the predicted information is false;

[0096] In order to further consider the semantic consistency between images and texts, a consistency loss formula is defined to calculate the distance between the modality fusion result and the semantic consistency result as follows:

[0097] L consistency =||FS|| 2

[0098] F represents the fused features, S represents the joint semantic consistency detection result, and L consistency Represents the consistency loss function.

[0099] The beneficial effect of the above scheme is that by calculating the distance between the multimodal fusion feature F and the semantic consistency detection result S to establish a consistency loss function, on the one hand, it further captures the degree of semantic matching between text and image. When F and S differ greatly, it indicates that the consistency information between the two modalities is not fully captured during the fusion process. On the other hand, it provides additional constraints for the model to prevent the model from relying too much on F or S to prevent overfitting.

[0100] Finally, combining the above two parts can get the final loss function;

[0101] L=α·L cross_entropy +β·L consistency

[0102] Where L represents the final loss function and α+β=1; by minimizing the loss function, the model will learn how to make better predictions given the input.

[0103] In view of the limitations of the prior art in simultaneously processing fine-grained and coarse-grained multimodal information, the present invention proposes a local and global feature extraction method that can effectively collect multimodal data from different granularity levels. This method realizes the comprehensive processing of multimodal information by extracting fine-grained and coarse-grained features of multimodal data respectively. In view of the problem of cross-modal semantic association modeling, the present invention provides a global-local semantic consistency joint modeling method that can effectively capture the complex semantic association relationship between image modality and text modality, thereby improving the semantic understanding ability of cross-modal data. By combining the deep fusion results of multimodal features with global-local semantic consistency analysis, the present invention significantly improves the accuracy of multimodal false information detection, enhances the robustness and generalization ability of the model in complex false news scenarios, and is suitable for the fields of multimodal data processing and false information detection.

[0104] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A multimodal false information detection method based on semantic consistency, characterized by: The following steps are involved: Step 1: Obtain news multimodal data from social networks and construct a training data set, where each data in the training data set includes news multimodal data and a label of whether it is false information, and the news multimodal data includes text data and image data; perform word segmentation on the text data; Step 2: Construct a multimodal false information detection model, which includes a global feature extractor LSTM and a local feature extractor BERT for text, and a global feature extractor CNN and a local feature extractor Swin Transformer for images, which are used to extract text global features V g and text local features V l , image global feature U g and the local image feature U l ; Step 3: Use the improved cosine distance to perform local semantic consistency detection on the extracted text local features and image local features, and use the Euclidean distance to perform global semantic consistency detection on the text global features and image global features. Combine the local semantic consistency and global semantic consistency detection results, and generate a joint semantic consistency result S through a dynamic fusion coefficient; Step 4: By assigning different weights, the text features and image features are effectively fused to generate fused features F representing multimodal information; Step 5: Concatenate and fuse the joint semantic consistency result S with the multimodal fusion feature F, and input the fused result G into the two-layer perceptron to obtain the feature representation H2 after the two-layer perceptron mapping processing, and then input the feature representation H2 into the classification layer to obtain the false information discrimination result p c , p c is the probability that the predicted information is false, using p c Construct a loss function and use the training data set in step 1 to train the false information detection model.

2. The multimodal false information detection method based on semantic consistency according to claim 1 is characterized in that: In the step 2: The extracting of the global features of the text specifically includes: traversing the text forward and backward through the global feature extractor LSTM to extract the global features V of the text g , the global text feature includes the dependency relationship between words in the text; The method of extracting the global features of the image specifically includes: using a global feature extractor CNN to obtain the global features U of the image through convolution, pooling, and full connection layer operations g ; The method of extracting local features of text specifically includes: encoding text information word by word through the local feature extractor BERT, extracting the representation of each word in the context, capturing the local information of each word in the context through the attention mechanism of BERT itself, and obtaining the local feature V of the text. l , where n is the number of words obtained after the word segmentation processing in step 1, the phrase-level information of each word is obtained through convolution kernels of different sizes, t represents the word obtained after the word segmentation processing in step 1, and v represents the local features of the word after word segmentation extracted after BERT processing. The specific formula is as follows: V l ={v1…v n }=BERT({t1…t n }) The method of extracting local features of an image specifically includes: dividing the input image into local image blocks of fixed size by using a local feature extractor Swin Transformer, applying the sliding window mechanism of Swin Transformer itself in the local image blocks, extracting features u of the local image blocks one by one, and combining the features extracted from each local image block to obtain the local feature U of the image. l , m represents the number of local image blocks, SWIN-T represents the Swin Transformer model, and e represents the local image block. The specific formula is as follows: HE l {u1...u m }=SWIN-T({e1…e m })。 3. The multimodal false information detection method based on semantic consistency according to claim 2 is characterized in that: The step three specifically includes: For each local image block feature u i , the local features v of each word extracted in step 2 j and the feature u of the local image block i The local semantic consistency score is calculated using the improved cosine theorem as follows: Take the maximum value of the local semantic consistency score as the semantic consistency result S of each local image block i , specifically expressed as: The local text image semantic consistency result S is obtained according to the following formula l (U l ,V l ): The text feature vector V is calculated as follows g and image feature vector U g The Euclidean distance in space is used to measure the consistency between modalities, and the weight matrix W is combined to adjust the consistency measurement result to obtain the global consistency information S g (V g ,U g ), and use the Relu activation function to ensure that the consistency result value is non-negative: The joint semantic consistency result S is calculated according to the following formula. The dynamic fusion coefficient λ is used to fuse the global semantic consistency result and the local semantic consistency result. S=λ*S l (V l ,U l )+(1-λ)*S g (V g ,U g )。 4. The multimodal false information detection method based on semantic consistency according to claim 3 is characterized by: The step 4 specifically includes: The global image feature U extracted in step 2 is g and text global feature V g Perform linear transformation and get U′ g and V′ g , achieving the purpose of mapping image and text modalities to the same dimension; The image modality and text modality are calculated through the self-attention mechanism to obtain the corresponding image and text attention weight matrices; according to the following formula, the weight matrix is ​​used to perform weighted calculation on the corresponding modality, and the image features and text features obtained by weighted calculation are fused: F=comcat(A U′ U′ g ,A V′ V′ g ) F represents the fused features, A U′ and A V′ Respectively represent the image and text attention weight matrices after being processed by the attention mechanism.

5. The multimodal false information detection method based on semantic consistency according to claim 4 is characterized in that: The step five specifically includes: The joint semantic consistency result S obtained in step 3 is fused with the multimodal fusion feature F in step 4. The specific operation for information discrimination is as follows: the joint semantic consistency result S is concatenated and fused with the multimodal fusion feature F, and the fused result G is input into the two-layer perceptron to obtain the feature representation H2 after the mapping processing of the two-layer perceptron. The feature representation H2 is then input into the classification layer for classification to obtain the false information discrimination result p. c , p c is the probability that the predicted information is false; G = concat(F,S) H2=ReLU(W2·ReLU(W1·G+b1)+b2) P c =σ(W o ·H2+b o ) W1,W2,,W o , represents three weight matrices, b1, b2, b o Indicates 3 bias values; Define the cross entropy loss function and the semantic consistency loss function, and use these two loss functions to construct a comprehensive cross entropy loss function for training a multimodal false information detection model, which specifically includes the following steps; Cross entropy loss function L cross_entropy The calculation formula is as follows: L cross_cntropy =-E[y·log p e +(1-y)·log(1-p c )] L cross_entropy is the cross entropy loss function, which represents the difference between the prediction and the true value. E represents the expected value, which means that the average loss is calculated for the data in the entire training data set. y is the true label. Construct the consistency loss L consistency : L consistency =|F-S|| 2 Combining the above two parts, we get the final loss function L; L=α·L cross_entropy +β·L consistency Where L represents the final loss function, α+β=1; using the training set described in step 1, the multimodal false information detection model is trained with the goal of minimizing the loss function L.

Images