Multi-mode forged information detection method combined with domain information

Abstract

The invention discloses a multi-mode forged information detection method combined with domain information. The method comprises the following steps that the domain information is recognized and described, a prompt learning template is designed for a big language model, and the big language model generates a field label of news and explains news content according to the prompt learning template; interactive learning of domain information and multi-modal features, including feature coding: coding original texts, domain information and images; text-domain and image-domain interactive feature learning: learning text features integrated with domain knowledge and image features integrated with domain knowledge by using a common attention mechanism; importance learning of domain features: splicing features related to domain information together, inputting the spliced features into a domain gating neural network, and calculating the importance of the domain information as the weight of the domain features; and feature aggregation and classification: splicing all features, and inputting the spliced features into a classifier to complete classification prediction.

Description

Technical Field

[0001] The invention relates to a multimodal forged information detection method combining domain information. Background Art

[0002] As social media platforms are increasingly integrated into people's lives, they have become the main source of information for the public. However, fake news can also be spread quickly, and the content of fake news can easily confuse people and mislead people, which in turn affects their judgment and decision. Therefore, fake information detection methods are needed to identify the authenticity of news. Current fake information detection methods mainly face the following problems:

[0003] (1) In real life, social media covers a variety of different fields, such as politics, economy, military, health, society, etc. Existing methods for detecting fake information are mostly limited to one field, which is insufficient and cannot meet the needs.

[0004] (2) Some multi-domain forged information detection methods have the problem of scarce domain labels. It is costly and time-consuming to label domain labels using manual annotation methods.

[0005] (3) Because the importance of domain information contained in different news samples varies, existing fake information detection methods are unable to adjust the feature weights of the final classification prediction based on the importance of domain information.

[0006] Therefore, the present invention is just produced based on the above shortcomings. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a multimodal forged information detection method combined with domain information. When integrating bimodal forged information detection, domain information is taken into account, and it can be applied to fake news detection in different fields. GPT3 is used to annotate domain labels to improve annotation efficiency. Different weights are given to domain information features according to the different importance of domain information of different news, thereby improving the adaptability of the model.

[0008] The present invention is achieved through the following technical solutions:

[0009] A multimodal forged information detection method combining domain information, characterized in that it comprises the following steps:

[0010] S1. Identify and describe domain information. Design a prompt learning template for the large language model. The large language model generates the domain label of the news and the explanation of the news content based on the prompt learning template.

[0011] S2, interactive learning of domain information and multimodal features, this step includes S21, feature encoding, encoding the original text, domain information and image;

[0012] S22, text-domain and image-domain interactive feature learning, using the common attention mechanism to learn text features that incorporate domain knowledge and image features that incorporate domain knowledge;

[0013] S23, learning the importance of domain features, splicing the features related to the domain information together, and inputting the spliced ​​features into the domain gated neural network, calculating the importance of the domain information as the weight of the domain feature;

[0014] S3, feature aggregation and classification, concatenate all features and input them into the classifier to complete classification prediction.

[0015] The multimodal forged information detection method combined with domain information as described above is characterized in that: in step S21, the news text and the domain information are encoded using Transformer to obtain the encoded text features respectively. and domain information features , use the Resnet network to encode the image and obtain the encoded image features .

[0016] The multimodal forged information detection method combining domain information as described above is characterized in that: in step S22, a linear layer with shared weights is first used to transform the text features , Field Information Characteristics and image features Mapped to the same feature space, the text features, domain information features and image features after mapping are obtained. The calculation formula is:

[0017] =

[0018] =

[0019] =

[0020] in represents the use of a linear layer with shared weights, Represents the text features obtained after mapping changes, Represents the image features obtained after mapping changes, Represents the domain information features obtained after mapping changes;

[0021] Taking the domain information feature as the query and the text feature as the key and value, the query matrix of the domain information feature, the key matrix of the text information feature, and the value matrix of the text information feature are calculated as follows:

[0022] =

[0023] =

[0024] =

[0025] in represents the linear transformation of the query matrix, The query matrix representing the characteristics of domain information, represents the linear transformation of the key matrix, The key matrix representing the features of text information, represents a linear transformation of the value matrix, A value matrix representing the characteristics of text information;

[0026] The formula for calculating the interactive features of text-domain information is:

[0027] =( )

[0028] Where H represents the number of attention heads, h represents the head of the attention mechanism, and d represents the feature dimension. Represents a text key, represents the linear transformation of the output, Represents the text-domain information interaction characteristics;

[0029] Taking the domain information features as the query and the image features as the key and value, the query matrix of the domain information features, the key matrix of the image features, and the value matrix of the image features are calculated as follows:

[0030] =

[0031] =

[0032] =

[0033] in represents the linear transformation of the query matrix, The query matrix representing the characteristics of domain information, represents the linear transformation of the key matrix, The key matrix representing the image features, represents a linear transformation of the value matrix, A matrix of values ​​representing image features;

[0034] The formula for calculating the interactive features of image-domain information is:

[0035] =( )

[0036] Where H represents the number of attention mechanism heads, h represents the head of the attention mechanism, represents the image key, represents the linear transformation of the output, Represents the image-domain information interaction feature.

[0037] The multi-modal forged information detection method combining domain information as described above is characterized in that: in step S23, , and Splicing, the spliced ​​features are input into the gated neural network. The output of the gated neural network is denoted as a, which represents the importance of domain information. The calculation formula is:

[0038] a=Softmax(G( )

[0039] in represents the parameters of the gated neural network, represents the concatenation operation, and G represents the domain gate.

[0040] The multimodal forged information detection method combined with domain information as described above is characterized in that step S3 comprises:

[0041] S31. Feature aggregation, assigning weights to domain features to obtain domain features with weights , the calculation formula is: =a* ,Bundle , , , , Splice them together to get feature v;

[0042] S32, feature classification, input feature v into the classifier to make the final classification prediction.

[0043] The multimodal forged information detection method combining domain information as described above is characterized in that the classifier used in step S32 is a Softmax classifier.

[0044] The multimodal forged information detection method combined with domain information as described above is characterized in that the large language model used in step S1 is GPT3.

[0045] Compared with the prior art, the present invention has the following advantages:

[0046] 1. The large language model is combined with the small language model. The large language model provides guidance for the small language model, and the small language model learns features and classifications under the guidance of the large language model. The large language model can be used to label data simply and efficiently, providing the advantage of interpretability, while avoiding the hallucinations of the large language model during the reasoning process and taking advantage of the strong feature extraction capability of the small language model. Therefore, in the present invention, GPT3 is used to provide labels and explanations for the domain information of the news, and then Transformer is used to encode the generated domain information and news text, and the Resnet network encodes the images of the news.

[0047] 2. Learned the correlation between domain information and images and texts. Using the joint attention mechanism, we learned the interaction between image and text modalities and domain information, so that domain knowledge can be integrated into feature learning.

[0048] 3. Dynamically and adaptively assign weights to domain information description features. The present invention splices domain information features, text-domain interaction features, and image-domain interaction features together, inputs them into the domain gated neural network, calculates the importance of domain information, and assigns weights to domain features in the subsequent feature aggregation process. When domain information is important, a larger weight is assigned; when domain information is unimportant, a smaller weight is assigned. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is the principle diagram of the present invention;

[0050] Figure 2 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described below in conjunction with the accompanying drawings:

[0052] like Figure 1 and Figure 2 As shown, a multimodal forged information detection method combining domain information includes the following steps:

[0053] S1. Identify and describe domain information. Design a prompt learning template for the large language model. The large language model generates the domain label of the news and the explanation of the news content based on the prompt learning template.

[0054] S2, interactive learning of domain information and multimodal features, this step includes S21, feature encoding, encoding the original text, domain information and image;

[0055] S22. Learning interactive features between text-domain and image-domain, using a common attention mechanism to learn text features that incorporate domain knowledge and image features incorporating domain knowledge ;

[0056] S23, learning the importance of domain features, splicing the features related to the domain information together, and inputting the spliced ​​features into the domain gated neural network, calculating the importance of the domain information as the weight of the domain feature;

[0057] S3, feature aggregation and classification, concatenate all features and input them into the classifier to complete classification prediction.

[0058] The large language model used in step S1 is GPT3. The prompt learning template designed for GPT3 is: "Pleaser recognize the domain area of ​​the given news. Then, use the domain knowledge to understand the news and describe the news from the perspective of the domain area." In this way, GPT3 will generate the domain label of the news and explain the news content from the perspective of domain knowledge based on the prompt learning template.

[0059] In step S21, Transformer is used to encode the news text and domain information to obtain the encoded text features. and domain information features , use the Resnet network to encode the image and obtain the encoded image features .

[0060] In step S22, in order to reduce the domain information features , text features , image encoding features We first use a linear layer with shared weights to transform the text features , Field Information Characteristics and image features Mapped to the same feature space, the text features, domain information features and image features after mapping are obtained. The calculation formula is:

[0061] =

[0062] =

[0063] =

[0064] in represents the use of a linear layer with shared weights, Represents the text features obtained after mapping changes, Represents the image features obtained after mapping changes, Represents the domain information features obtained after mapping changes;

[0065] Because text modality and image modality are related to domain information, a joint attention mechanism is needed to capture the relationship between text and domain information, image and domain information features, and obtain text-domain and image-domain interaction features. In order to obtain text-domain interaction features, domain information features are used as queries, text features are used as keys and values, and the query matrix of domain information features, the key matrix of text information features, and the value matrix of text information features are calculated as follows:

[0066] =

[0067] =

[0068] =

[0069] in represents the linear transformation of the query matrix, The query matrix representing the characteristics of domain information, represents the linear transformation of the key matrix, The key matrix representing the features of text information, represents a linear transformation of the value matrix, A value matrix representing the characteristics of text information;

[0070] The formula for calculating the interactive features of text-domain information is:

[0071] =( )

[0072] Where H represents the number of attention heads, h represents the head of the attention mechanism, and d represents the feature dimension. Represents a text key, Represents a text value, represents the linear transformation of the output, Represents the text-domain information interaction characteristics;

[0073] Similarly, you can refer to The method uses the domain information feature as the query and the image feature as the key and value to calculate the query matrix of the domain information feature, the key matrix of the image feature, and the value matrix of the image feature as follows:

[0074] =

[0075] =

[0076] =

[0077] in represents the linear transformation of the query matrix, The query matrix representing the characteristics of domain information, represents the linear transformation of the key matrix, The key matrix representing the image features, represents a linear transformation of the value matrix, A matrix of values ​​representing image features;

[0078] The formula for calculating the interactive features of image-domain information is:

[0079] =( )

[0080] Where H represents the number of attention mechanism heads, h represents the head of the attention mechanism, represents the image key, represents the image value, represents the linear transformation of the output, Represents the image-domain information interaction feature.

[0081] In step S23, , and Then, the concatenated features are input into the gated neural network. The output of the gated neural network is recorded as a, which represents the importance of the domain information and serves as the weight of the domain feature. The calculation formula is:

[0082] a=Softmax(G( )

[0083] in represents the parameters of the gated neural network, represents the concatenation operation, and G represents the domain gate.

[0084] In this method, step S3 includes two parts: feature aggregation and feature classification.

[0085] S31. Feature aggregation, assigning weights to domain features to obtain domain features with weights , the calculation formula is: =a* ,Bundle , , , , Splice them together to get feature v;

[0086] S32, feature classification, input feature v into the classifier to make the final classification prediction. The classifier used in step S32 is a Softmax classifier.

[0087] In general, the present invention takes into account domain information when integrating dual-modal forged information detection, and can be applied to fake news detection in different fields. Using GPT3 to label domain labels can avoid the problem of low efficiency of manual labeling, while providing interpretability; the text modality and image modality use the common attention mechanism to learn domain-related interactive information; the importance of domain information is learned, and different weights are given to domain information features according to the different importance of domain information of different news, thereby improving the adaptability of the model.

Claims

1. A multimodal forged information detection method combining domain information, characterized in that: The following steps are involved: S1. Identify and describe domain information. Design a prompt learning template for the large language model. The large language model generates the domain label of the news and the explanation of the news content based on the prompt learning template. S2, interactive learning of domain information and multimodal features, this step includes S21, feature encoding, encoding the original text, domain information and image; S22, text-domain and image-domain interactive feature learning, using the common attention mechanism to learn text features that incorporate domain knowledge and image features that incorporate domain knowledge; S23, learning the importance of domain features, splicing the features related to the domain information together, and inputting the spliced ​​features into the domain gated neural network, calculating the importance of the domain information as the weight of the domain feature; S3, feature aggregation and classification, concatenate all features and input them into the classifier to complete classification prediction.

2. The multimodal forged information detection method combining domain information according to claim 1 is characterized in that: In step S21, the news text and domain information are encoded using Transformer to obtain the encoded text features. and domain information features , use the Resnet network to encode the image and obtain the encoded image features .

3. The multimodal forged information detection method combining domain information according to claim 2 is characterized in that: In step S22, the text features are firstly transformed into , Field Information Characteristics and image features Mapped to the same feature space, the text features, domain information features and image features after mapping are obtained. The calculation formula is: = ； = ； = ； in represents the use of a linear layer with shared weights, Represents the text features obtained after mapping changes, Represents the image features obtained after mapping changes, Represents the domain information features obtained after mapping changes; Taking the domain information feature as the query and the text feature as the key and value, the query matrix of the domain information feature, the key matrix of the text information feature, and the value matrix of the text information feature are calculated as follows: = ； = ； = ； in represents the linear transformation of the query matrix, The query matrix representing the characteristics of domain information, represents the linear transformation of the key matrix, The key matrix representing the features of text information, represents a linear transformation of the value matrix, A value matrix representing the characteristics of text information; The formula for calculating the interactive features of text-domain information is: =（ ） ； Where H represents the number of attention heads, h represents the head of the attention mechanism, and d represents the feature dimension. Represents a text key, represents the linear transformation of the output, Represents the text-domain information interaction characteristics; Taking the domain information features as the query and the image features as the key and value, the query matrix of the domain information features, the key matrix of the image features, and the value matrix of the image features are calculated as follows: = ； = ； = ； in represents the linear transformation of the query matrix, The query matrix representing the characteristics of domain information, represents the linear transformation of the key matrix, The key matrix representing the image features, represents a linear transformation of the value matrix, A matrix of values ​​representing image features; The formula for calculating the interactive features of image-domain information is: =（ ） ； Where H represents the number of attention mechanism heads, h represents the head of the attention mechanism, represents the image key, represents the linear transformation of the output, Represents the image-domain information interaction feature.

4. The multimodal forged information detection method combining domain information according to claim 3 is characterized in that: In step S23, , and Splicing, the spliced ​​features are input into the gated neural network. The output of the gated neural network is denoted as a, which represents the importance of domain information. The calculation formula is: a=Softmax(G( )); in represents the parameters of the gated neural network, represents the concatenation operation, and G represents the domain gate.

5. The multimodal forged information detection method combining domain information according to claim 4 is characterized in that: The step S3 comprises: S31. Feature aggregation, assigning weights to domain features to obtain domain features with weights , the calculation formula is: =a* ,Bundle , , , , Splice them together to get feature v; S32, feature classification, input feature v into the classifier to make the final classification prediction.

6. The multimodal forged information detection method combining domain information according to claim 5 is characterized in that: The classifier used in step S32 is a Softmax classifier.

7. The multimodal forged information detection method combining domain information according to claim 1 is characterized in that: The large language model used in step S1 is GPT3.

Images