False news detection method based on image-text common feature and specific feature adaptive selection fusion

Abstract

The invention relates to a false news detection method based on image-text common feature and specific feature adaptive selection fusion, and relates to the field of false news detection. The method mainly comprises the steps of providing a deep learning method for performing false news detection by utilizing common features (consistency features) among news modes and specific features (inconsistency features) of each mode, providing a common feature extraction module based on a common attention mechanism, and promoting feature complementation and interactive understanding among the modes. Capturing consistency characteristics between modals; designing a single-mode feature filtering strategy, filtering redundant information and useless noise in single-mode features, and extracting inconsistent features of each mode; and the use of each feature is adaptively adjusted through the cosine similarity score, so that the accuracy of identifying false news by the model is effectively improved. The method is suitable for a false news detection task in an actual scene, and has an actual application value in an actual life scene.

Description

Technical Field

[0001] The present invention relates to the field of false news detection, and in particular to a false news detection method based on adaptive selection and fusion of common features of images and texts and unique features. Background Art

[0002] Fake news detection technology is a key technology that aims to deal with the increasingly serious problem of fake news dissemination. In the digital age of information explosion, the application of fake news detection technology has become particularly important. This technology analyzes and identifies the authenticity of news content by utilizing advanced natural language processing, image processing and multimodal learning algorithms, and has the characteristics of accurate detection, strong adaptability and real-time response. The application prospects of fake news detection technology are broad, covering multiple fields such as media, social platforms and justice. In the media field, fake news detection technology can help news organizations ensure the accuracy of reports, prevent the spread of fake news, and maintain the credibility of the news industry. On social platforms, this technology can effectively filter out false content and ensure the quality of information obtained by users. In the judicial field, fake news detection technology can provide strong technical support for legal institutions to help identify and review the authenticity of evidence. However, with the development of technology, the generation of fake news has become more complex and diverse, bringing new challenges to fake news detection technology. Therefore, it is necessary to continue to study and improve fake news detection technology to provide more reliable information authenticity verification solutions to cope with the increasingly complex means of generating false information.

[0003] In the early days, fake news detection tasks only focused on a single modality in the news, such as image modality or text modality. The paper "J.Ma, W.Gao, P.Mitra, S.Kwon, BJJansen, KFWong, M.Cha, Detecting rumors from microblogs with recurrent neural networks, Proceedings of the 25th International Joint Conference on Artificial Intelligence, 2016, pp.3818-3824" input the sentences in the news into the recurrent neural network RNN, and used the hidden layer vector to represent the news information, which was the first time that deep learning technology was applied to the fake news detection task. However, most of the current news contains multimodal information such as text and images, and the detection method for a single modality is not enough to learn cross-modal features. To solve this problem, the paper "S. Singhal, RR Shah, T. Chakraborty, P. Kumaraguru, S. Satoh, Spotfake: A multi-modal framework for fake news detection, IEEE fifth international conference on multimedia bigdata, 2019, pp. 39-47" used the pre-trained models BERT and VGG19 to extract text and image features respectively, and then combined the two features for fake news detection. Subsequently, the researchers found that both single-modal features and multi-modal fusion features have positive contributions to fake news detection. The paper "Y.Chen, D.Li, P.Zhang, J.Sui, Q.Lv, L.Tun, L.Shang, Cross-modalAmbiguity Learning for Multimodal Fake News Detection, Proceedings of the ACMweb conference, 2022, pp.2897-2905" proposes a cross-modal alignment auxiliary task, uses KL divergence to measure the matching degree between modalities, and weights the matching score to single modal features and multimodal fusion features.The document "Q. Ying, X. Hu, Y. Zhou, Z. Qian, D. Zeng, S. Ge, Bootstrapping Multi-View Representations for Fake News Detection, Proceedings of the AAAI conference on Artificial Intelligence, 2023, pp. 5384-5392." proposes to decompose unimodal and multimodal features through unimodal prediction and cross-modal consistency learning, and then adaptively reweight and guide to achieve fake news detection. However, not all information in unimodal features is effective for the fake news detection task, and too much attention is paid to multimodal fusion features, that is, the common features (consistency features) between modalities, ignoring the contribution of the unique features (inconsistency features) of each modality to the detection task, and the use of each modal feature of news is not comprehensive enough. In contrast, the present invention is specially designed to capture the common features between modalities and the unique features of each modality to promote feature complementarity enhancement between modalities and the effective use of unimodal features. Taking into account the problems of practical applications, the present invention aims to improve the performance of fake news detection tasks in social media communication scenarios, and proposes a fake news detection method based on adaptive selection and fusion of common features and unique features of images and texts. Summary of the invention

[0004] The present invention proposes a fake news detection method based on adaptive selection and fusion of common features and unique features of images and texts, which can realize the authenticity identification of news spread on social media. The overall fake news detection framework includes five structures: a feature encoder based on a pre-trained model, a common feature extraction module based on a common attention mechanism, an image-specific feature extraction branch and a text-specific feature extraction branch based on a unimodal feature filtering strategy, and a fake news classifier that adaptively adjusts the feature usage based on the cosine similarity score. Figure 1 As shown in Figure 1, the specific process of the fake news detection framework is as follows: the text and pictures of the news are sent to the pre-trained model encoder in pairs (the text length does not exceed 300 and the picture size is 224×224), and the semantic features R of the text modality are encoded using the BERT pre-trained model. T , using the Swin-T pre-trained model to encode the semantic features of the image modality R I , use the CLIP multimodal pre-trained model to encode the global features C of both text and image modalities T and C I ; Semantic feature R T and R I The common feature R between the enhanced modes is obtained through the common feature extraction module M(For ease of understanding, the following text uses consistency features instead of common features); at the same time, R T and R I The inconsistency score vectors are calculated by the text-specific feature extraction branch and the image-specific feature extraction branch to obtain the unique features R of the text and image. T_incon and R I_incon (For ease of understanding, the following text uses inconsistent features instead of unique features); global features C T and C I is used to calculate the cosine similarity score, obtain the cosine similarity score normalized between 0 and 1, and use the cosine similarity score to weight R T_incon , R I_incon and R M , adaptively adjust the use of each feature, and finally send it to the fake news classifier for news true and false classification. In previous methods, more attention was paid to the consistency features between modalities, and less targeted processing of single-modal features. In the present invention, a single-modal feature filtering strategy is designed to extract the inconsistent features of each modality in a targeted manner, and at the same time, through the cosine similarity score, the inconsistent features of each modality and the consistent features between modalities are adaptively involved in the fake news detection task. It mainly includes three contents:

[0005] (1) Propose a common feature extraction module based on the joint attention mechanism;

[0006] (2) Propose a unimodal feature filtering strategy;

[0007] (3) A fake news classifier is proposed that uses adaptively adjusted features based on cosine similarity scores.

[0008] The specific contents are as follows:

[0009] (1) A common feature extraction module based on the co-attention mechanism is proposed: three pairs of co-attention components based on the co-attention mechanism are constructed (each pair of components contains two co-attention modules), which fuse the features of the two modalities of text and image and enhance the multimodal fusion features to capture the consistent features between the modalities.

[0010] In the joint attention mechanism of the present invention, a pair of joint attention components includes two joint attention modules, the input feature dimension of each joint attention module is 512, and each pair of joint attention components internally shares parameters. Figure 2 As shown, each joint attention module consists of a multi-head attention layer, a feed-forward network layer, an average pooling layer, and two normalization layers.

[0011] The common feature extraction module consists of three pairs of common attention components and TFN fusion strategy, such as Figure 3As shown in Figure 2, the joint attention component 1 is used to fuse the semantic features R of text and image. T and R I , the joint attention component 2 is used to pass the image semantic features R I Enhanced fusion features, the common attention component 3 is used to pass the text semantic feature R T Enhance the fusion features, and finally obtain the consistent feature R between the modalities through the TFN fusion strategy M Specifically, R T and R I As the input of the joint attention component 1, the multi-head attention layer of each joint attention module has H = 8 heads. First, R T and R I They are mapped to the same feature space through linear transformation respectively, and each head h can be expressed as:

[0012]

[0013] in and is the learnable weight matrix corresponding to each head, n represents the length of the text, p = 49 represents the number of blocks of the image, and d w and d i Represent the semantic feature dimensions of text and image respectively, d k =64 represents the dimension of query(Q), key(K) and value(V) vectors of each head in the multi-head attention layer. Then the attention weight of a single attention head is calculated, which can be expressed as:

[0014]

[0015] in Calculate the attention output of a single attention head through a single attention weight:

[0016]

[0017] The attention outputs of the H attention heads in the multi-head attention layer are concatenated and linearly transformed to generate the final output features of the multi-head attention layer:

[0018]

[0019] in and is a linear transformation matrix. The feature representation is updated through residual connection and layer normalization. Finally, the updated feature representation is passed through the fully connected layer and then layer normalized and average pooled into a feature vector as the output of each common attention module. The two outputs are concatenated to obtain the multimodal fusion feature r of the common attention component 1. it :

[0020]

[0021] r it =concat(r ci ,r ct ) (7)

[0022] Where LayerNorm represents layer normalization, and “” represents average pooling.

[0023] Use R I and R T The multimodal fusion feature r is processed by the joint attention component 2 and the joint attention component 3. it For feature enhancement, the calculation steps are consistent with the above-mentioned common attention component 1:

[0024]

[0025] The TFN strategy is used to fuse two multimodal fusion features after feature enhancement. it_t and r it_i First, we first analyze r it_t and r it_i Use 1 to expand the dimension and then find the Cartesian product:

[0026] R M =Cartesian[concat(r it_t ,1),concat(r it_i ,1)] (9)

[0027] Cartesian means calculating Cartesian product. Through the above steps, we finally get the multimodal fusion feature R that integrates each modal feature and undergoes feature enhancement. M , as a consistent feature between modalities, and participates in fake news classification.

[0028] (2) A unimodal feature filtering strategy is proposed: a unimodal feature processing flow is constructed to eliminate the parts of a single modality semantic feature that are highly similar to another modality, capture the inconsistent features in the single modality features, help effectively utilize the unimodal features and filter out useless noise and redundant information.

[0029] The unimodal feature filtering strategy in the present invention is applied to Figure 1 The image-specific feature extraction branch and the text-specific feature extraction branch are shown in FIG. In the image-specific feature extraction branch, one of the inputs is the image semantic feature R extracted by the Swin-T model. I , and the other input comes from the image feature r weighted by text attention extracted by the common attention component 1 of the common feature extraction module described in (1)ci , R I With r ci Multiply to get the image similarity matrix:

[0030]

[0031] Then the image similarity matrix The image similarity vector is obtained by summing up the columns, and then the image similarity vector is normalized by the softmax function to obtain the image similarity score vector:

[0032]

[0033] Where ∑ represents the sum of columns.

[0034] Flip the image similarity score vector, that is, subtract 1 from the image similarity score vector to obtain the image inconsistency score vector, and weight the image inconsistency score vector to the image semantic feature R I , calculate the image inconsistency features. Finally, the acquired image inconsistency features are filtered through the self-attention mechanism, and the final inconsistency features R of the image modality are output I_incon :

[0035] r I_incon =1-r is , R I_incon =FFN(SA(PM(r i_incon ,R I ))) (12)

[0036] PM represents the dot product operation, and SA represents the self-attention mechanism. The higher the image inconsistency score at the corresponding position, the less attention is paid to this area of ​​the image when extracting consistent features, indicating inconsistency. The obtained image inconsistency features may contain some useless information and interference noise, so they are filtered through the self-attention mechanism.

[0037] In the text-specific feature extraction branch, the input feature is the text semantic feature R extracted by the BERT model. T The text feature r weighted by image attention extracted by the common attention component 1 of the common feature extraction module described in (1) ct , text inconsistency feature R T_incon The extraction process of is the same as the process of obtaining image inconsistency features by the image-specific feature extraction branch.

[0038] (3) A fake news classifier based on adaptively adjusted features using cosine similarity scores is proposed: a network is constructed to classify fake news. The input is the inter-modal consistency features and the inconsistency features of each modality weighted by the cosine similarity scores. The output is the predicted label of the news, i.e. true or false.

[0039] The fake news classifier in the present invention weights the inter-modality consistency feature R described in (1) by the cosine similarity score. M And the inconsistency feature R of the image and text described in (2) I_incon and R T_incon , adaptively adjust the contribution of each feature in the fake news detection task and guide the classifier learning. Specifically, Figure 1 As shown, the global feature C extracted by the CLIP model T and C I Calculate the cosine similarity score and normalize it between 0 and 1:

[0040]

[0041] Use the normalized cosine similarity score sim to weight the image inconsistency feature R I_incon and text inconsistency feature R T_incon , weight the inter-modal consistency feature R with 1 minus sim M :

[0042] R If =sim·R I_inc on, R Tf =sim·R T_inc on, R Mf =(1-sim)·R M (14)

[0043] Splicing feature R Tf , R If and R Mf , fed into the fake news classifier:

[0044] y'=classifier(concat(R If ,R Tf ,R Mf )) (15)

[0045] Where y' is the predicted label (0 or 1), classifier represents the fake news classifier, which consists of a two-layer fully connected network and a Relu activation function. The loss function used here is the binary cross entropy loss:

[0046] L BCE=-[ylog(y')+(1-y)log(1-y')] (16)

[0047] Where y represents the true label and y' represents the predicted label.

[0048] Compared with the prior art, the above scheme has at least the following significant effects:

[0049] 1. The present invention provides a fake news detection method based on adaptive selection and fusion of common features and unique features of images and texts. The common feature extraction module in the fake news detection framework can extract consistent features between modalities. The common feature extraction module in the present invention can fuse image semantic features and text semantic features through a common attention mechanism, and use the semantic features of each modality to enhance the fused features. Compared with existing methods, existing methods only consider simple fusion between features of each modality, and the learned cross-modal features are not enough or even cannot be learned. In this solution, after the features of each modality are fused, the features of each modality are reintroduced to enhance the fused features, which fully promotes feature interaction between modalities and can effectively help mutual understanding of information between modalities.

[0050] 2. The present invention provides a false news detection method based on adaptive selection and fusion of common features and unique features of images and texts. The text-specific feature extraction branch and the image-specific feature extraction branch in the false news detection framework can capture the inconsistency features of the corresponding modalities. The unimodal feature filtering strategy in the present invention is applied to the text-specific feature extraction branch and the image-specific feature extraction branch, and the inconsistency score vector can be calculated for the unimodal feature, so as to weight the corresponding unimodal semantic features to capture the inconsistency features of a single modality, and use the self-attention mechanism to filter useless noise. Compared with the existing methods, most of the existing methods focus on the consistency features between modalities, and the processing of unimodal features is not targeted. In this solution, attention is paid to the consistency features between modalities and the inconsistency features of each modality at the same time, and a unimodal feature filtering strategy is designed to extract the inconsistency features in a targeted manner, filter the useless noise and redundant information in the unimodal features, and help to effectively utilize the unimodal features. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a general schematic diagram of the fake news detection framework of the present invention.

[0052] Figure 2 Schematic diagram of a single joint attention module of the present invention

[0053] Figure 3 Schematic diagram of the common feature extraction module of the present invention DETAILED DESCRIPTION

[0054] The present invention is a false news detection method based on adaptive selection and fusion of common features and unique features of images and texts. The specific implementation methods of the present invention are exemplified below using the Weibo-21 dataset as a training and test dataset. It should be understood by those skilled in the art that these implementation methods are only used to explain the technical principles of the present invention and are not intended to limit the scope of application of the present invention.

[0055] The development language is Python. The deep learning framework is Pytorch. The specific steps are as follows:

[0056] Step 1: Dataset preparation and preprocessing. Weibo-21 is a Chinese dataset containing 4640 real news and 4487 fake news. It is divided into training set and test set in a ratio of 9:1, that is, the training set contains 8214 news data and the test set contains 913 news data. First, filter out news without images or text descriptions; second, match the text and image of each news one by one; third, preprocess the text and image, convert the image format into a tensor, and establish a mapping between the image name and the corresponding image tensor; fourth, clean the text content by filtering special symbols, aligning lengths, and other operations, and generate a text index.

[0057] Step 2: Pre-training model to extract text and image features. Load the BERT pre-training model to extract text features, use the "bert-base-chinese" model on the Weibo-21 Chinese dataset (the "bert-base-uncased" model should be used for the English dataset), segment the text and generate word vectors, which can extract rich contextual semantic information. The last layer output is used as the text semantic feature R T Load the Swin-T pre-trained model to extract image features, use the "swin-base-patch4-window7-224" model to split the image into 49 small patches, capture the image context semantic features through the moving window mechanism, and use the output of the last Transformer layer as the image semantic feature R I Load CLIP multimodal pre-trained model to extract global features of image and text modalities simultaneously I and C T , embedding of text and images is achieved through multimodal feature encoding, thereby mapping them into a unified mathematical space.

[0058] Step 3: Construct a common feature extraction module. The common feature extraction module consists of three pairs of common attention components and TFN fusion strategy. Each pair of common attention components consists of two common attention modules, such as Figure 3 As shown. First, the image semantic feature R described in step 2 is Iand text semantic features R T Linearly transform to feature dimension 512, then input into common attention component 1 to obtain image feature r weighted by text attention weight ci and the text feature r weighted by the image attention weight ct , concatenate the two features to get the fusion feature r it . Second, r it and R I Input the common attention component 2 and concatenate the two features output to obtain the fused feature r enhanced with image semantic features. it_i ; r it and R T Input the common attention component 3 and concatenate the two features output to obtain the fused feature r enhanced with text semantic features. it_t Third, the enhanced fusion feature r is fused through the TFN fusion strategy. it_i and r it_t Expand the dimensions by 1 respectively, and then calculate the Cartesian product to finally obtain the inter-modal consistency feature R M .

[0059] Step 4: Construct text-specific feature extraction branches and image-specific feature extraction branches. Figure 1 As shown in Figure 2, both the text-specific feature extraction branch and the image-specific feature extraction branch capture inconsistent features based on a unimodal feature filtering strategy. In the image-specific feature extraction branch, the dimension of the image semantic feature in step 1 is first linearly transformed to 512, and then combined with the image feature r weighted by the text attention weight in step 3. ci Multiply them to get an image similarity matrix with a dimension of 512×512, then sum the image similarity matrix by column to get the image similarity vector, and then normalize the image similarity vector through the softmax function to get the image similarity score vector, and flip the image similarity score vector by 1 minus the image similarity score vector to get the image inconsistency score vector, and use the image inconsistency score vector to weight the image semantic features with a feature dimension of 512, and filter out useless information and interference noise through the self-attention mechanism, and finally get the image inconsistency feature R I_incon The text-specific feature extraction branch extracts text inconsistency features R T_incon The process is consistent with the image-specific feature extraction branch.

[0060] Step 5: Construct a fake news classifier. The fake news classifier based on adaptively adjusting features using cosine similarity scores first uses the global features C of the image and text described in step 1. I and C TCalculate the cosine similarity score and normalize it between 0 and 1. The normalized cosine similarity score is sim. Use sim to weight the image inconsistency feature R described in step 4. I_incon and text inconsistency feature R T_incon , use (1-sim) to weight the inter-modal consistency feature R described in step 3 M , the weighted features are concatenated and input into the fake news classifier to identify fake news.

[0061] Step 6: Fake news detection model training. The model is trained using the binary cross entropy loss function, and the learning rate of the fake news detection model is set to 0.0001. At the same time, in order to make the pre-trained model better adapt to the fake news detection task, the learning rate of the BERT model and the Swin-T model is set to 0.00001. Set the batch size to 16, and take 16 news items from the training set described in step 1 in each batch (the news items contain corresponding text and images), perform content cleaning operations such as removing special symbols and aligning length on the text, and uniformly scale the images to a size of 224×224. Update the model parameters using the Adam optimizer. The epoch is set to 100. After each epoch is trained, the accuracy of the model on the test set described in step 1 is evaluated, and the model parameters with the highest accuracy on the test set are saved as the final model.

[0062] Step 7: Detect fake news in social media dissemination scenarios. For news from any social media platform that contains images and text content, first perform content cleaning operations such as removing special symbols and aligning lengths on the text, then scale the image to a size of 224×224 and convert it into a tensor. Input the preprocessed text and image into the pre-trained model to extract features, extract the inconsistency features of the text and image respectively through the text-specific feature extraction branch and the image-specific feature extraction branch, capture the consistency features between modalities through the common feature extraction module, then calculate the cosine similarity score and weight the obtained features, adaptively adjust the use of each feature, guide the classifier learning, and finally splice the features weighted by the cosine similarity score and input them into the fake news classifier for authenticity identification to obtain the predicted label (true or false) of the predicted news.

[0063] In summary, for the scenario of social media dissemination, the present invention designs a false news detection method based on the adaptive selection and fusion of common features of images and texts and unique features. For news disseminated on social media, the method has high detection accuracy, which meets the requirements of accurate identification of false news in actual complex scenarios to avoid causing greater losses. In addition, since the inconsistency features of each modality are introduced on the basis of focusing on the consistency features between modalities, the model makes more full use of the information of each modality of news.

[0064] Those skilled in the art will appreciate that the protection scope of the present invention is not limited to the specific embodiments described. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features. It should be noted that the technical solutions after the changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A fake news detection method based on adaptive selection and fusion of common features and unique features of images and texts, It is characterized in that The pre-trained model is used to obtain the semantic features and global features of multimodal news images and texts, and a common feature extraction module based on the common attention mechanism is constructed. This module integrates and complements the semantic features of the text modality and the image modality, extracts the common features between the image and the text (for ease of understanding, the common features are replaced by consistency features below), and combines the unimodal feature filtering strategy to calculate the inconsistency score vector for a single modality to extract the unique features of the text and image (for ease of understanding, the unique features are replaced by inconsistency features below). The cosine similarity score is used to adaptively integrate the consistency features and the inconsistency features, and all the features are sent to the classifier for false news classification, including: A feature encoder based on a pre-trained model, a common feature extraction module based on a common attention mechanism, an image-specific feature extraction branch and a text-specific feature extraction branch based on a unimodal feature filtering strategy, and a fake news classifier that adaptively adjusts feature usage based on cosine similarity scores; the feature encoder receives the text and image of the news and obtains the semantic features R of the text and image. T and R I , the global features C of text and image T and C I ; The common feature extraction module integrates the semantic features of text and image through the common attention mechanism R T and R I , get the fusion feature r it , and using R T and R I R it Perform information enhancement to obtain enhanced multimodal fusion features, that is, the consistency feature R between the modes M ; The text-specific feature extraction branch and the image-specific feature extraction branch calculate the inconsistency score vector r through the unimodal feature filtering strategy respectively t_incon and r i_incon , using r t_incon and r i_incon Weigh the semantic features R T and R I Get the inconsistency feature R T_incon and R I_incon ; Use the cosine similarity score to measure R M , R T_incon and R I_incon Weighting, adaptively adjust the use of each feature, and send them together to the fake news classifier for news classification; The common feature extraction module based on the common attention mechanism includes multiple pairs of common attention components and TFN fusion strategy. Each pair of common attention components includes two common attention modules, which help to fuse the semantic features of text and image and enhance the multimodal fusion features to obtain the consistency features between modalities. The image features r weighted by text attention are obtained through the common attention component 1. ci and the text feature r weighted by image attention ct , concatenate to get the fusion feature r it , the attention weighting and fusion operation of the joint attention component can be defined as: ci =CA(R I ,R T ), r ct =CA(R T ,R I ), r it =concat(r ci ,r ct ), where CA represents the joint attention module, and each pair of joint attention components shares parameters; the image semantic features R are used by joint attention component 2 and joint attention component 3 respectively. I and text semantic features R T For the fusion feature r it To enhance information, the process can be described as: it_i =concat[CA(R I ,r it ),CA(r it ,R I )],r it_t =concat[CA(R T ,r it ),CA(r it ,R T )]; Finally, based on the TFN fusion strategy, r it_i and r it_t Expand the dimension with 1 and calculate the Cartesian product to get the consistent feature R of the text and image. M , which can be expressed as: R M =Cartesian[concat(r it_i ,1),concat(r it_t ,1)], where Cartesian means calculating the Cartesian product.

2. According to claim 1, the false news detection method based on adaptive selection and fusion of common features and unique features of images and texts, It is characterized in that The unimodal feature filtering strategy used in the image-specific feature extraction branch and the text-specific feature extraction branch based on the unimodal feature filtering strategy is the same, that is, calculating the inconsistency score vector of the corresponding modality, using the inconsistency score vector to weight the semantic features of the corresponding modality, filtering redundant information and useless noise through the self-attention mechanism, and finally obtaining the inconsistency features of the corresponding modality, which specifically includes: The image-specific feature extraction branch uses the image semantic feature R I The image feature r obtained by the common feature extraction module with text attention weighting ci Multiply them to get the image similarity matrix, sum the image similarity matrix by column to get the image similarity vector, then normalize the image similarity vector through the softmax function to get the image similarity score vector, and then perform similarity flipping, that is, subtract 1 from the image similarity score vector to get the image inconsistency score vector r i_incon , the calculation process can be expressed as: i_incon =1-softmax(ΣR T T r ci ), where Σ represents the summation by column; the image inconsistency score vector is weighted with the image semantic feature, and the redundant information and useless noise are filtered through the self-attention mechanism to obtain the final image inconsistency feature R I_incon , which can be expressed as: R I_incon =FFN(SA(r i_incon ·R I )), where SA represents the self-attention mechanism and FFN represents the feed-forward network; The text-specific feature extraction branch has the same processing process as the image-specific feature extraction branch, except that the features processed are text semantic features R T The text feature r weighted by image attention obtained by the common feature extraction module ct , the resulting text inconsistency score vector is r t_incon , the final extracted text inconsistency feature is R T_incon .

3. According to claim 1, the false news detection method based on adaptive selection and fusion of common features and unique features of images and texts, It is characterized in that The fake news classifier based on the cosine similarity score adaptive adjustment feature uses the calculated cosine similarity score to weight the consistency feature between the modalities and the inconsistency feature of each modality, and classifies the fake news through the fake news classifier, specifically including: Through the global features C of text and image T and C I Calculate the cosine similarity and normalize it to between 0 and 1, expressed as: f s =[C T ·(C I ) T ] / [||C T ||·||C I ||],sim=(1+f s ) / 2; using sim to weight the inconsistency features of text and image, (1-sim) to weight the consistency features between modalities, it can be expressed as: R Tf =sim·R T_incon , R If =sim·R I_incon , R Mf =(1-sim)·R M ; Finally, the weighted features are concatenated and sent to the fake news classifier to obtain the predicted label y': y' = classifier[concat(R If ,R Tf ,R Mf )], where classifier represents the fake news classifier, which consists of two fully connected layers and Relu activation function.

Images