A multi-modal fact-checking method based on entity augmentation and stance checking

Abstract

The application discloses a kind of multi-modal fact checking methods based on entity enhancement and stand checking, comprising: 1, valuable clues are extracted from the statement containing text and image, such as text, image and its entity information, as preliminary feature;2, simulate the hierarchical reasoning process of human, extract the stand semantic reasoning feature by sharing prototype guided feature reconstruction, to check the consistency between different types of statements and evidence;3, construct cross-modal entity pair set, and design knowledge related reasoning strategy, to find the shortest semantic related path between each pair of entities in external knowledge graph;Fact checking network refines the distance and entity representation of knowledge enhancement by absorbing the supplementary background knowledge of all entities in this path.The application can learn effective semantic level and knowledge level statement representation by jointly simulating stand semantic reasoning feature and knowledge enhanced entity pair feature, thereby improving the accuracy and efficiency of multi-modal fact checking.

Description

Technical Field

[0001] This invention belongs to the field of multimodal fact-checking, specifically a multimodal fact-checking method based on entity augmentation and position checking. Background Technology

[0002] Fact-checking refers to assessing the veracity of statements in written or spoken language using retrieved evidence. The widespread dissemination of rumors on social media can have adverse effects. Given the sheer volume and rapid spread of new information, manual verification is increasingly inadequate. Therefore, there is a growing need for automated methods to check claims on social media platforms and promote truthful information. Most existing fact-checking research focuses solely on textual claims. These methods can be categorized into two types: 1) treating multiple pieces of evidence as a single piece and employing a textual implication model; 2) using specialized components to decompose the evidence for fine-grained fact-checking. With the development of multimedia technology, rumor spreaders tend to utilize both visual and textual content to attract more attention and accelerate dissemination. Therefore, multimodal fact-checking has become a new and attractive task. Compared to unimodal fact-checking, learning effective feature representations from heterogeneous multimodal data is more challenging but also provides rich complementary cues that can help detect the veracity of claims.

[0003] To accomplish this task, numerous multimodal fact-checking detectors have been proposed to explore cross-modal information interaction and fusion, thereby identifying anomalies in rumors. However, most of them merely capture basic semantic relationships between images and text at a simple feature level: these methods simply utilize connective operations, attention mechanisms, or auxiliary tasks to identify basic semantic relationships between visual and textual features and generate multimodal representations. Furthermore, the importance of fundamental higher-order knowledge and semantic relationships within multimedia content is often overlooked. Summary of the Invention

[0004] To address the shortcomings of the existing technologies, this invention proposes a multimodal fact-checking method based on entity augmentation and position checking. This method aims to improve the accuracy and efficiency of multimodal fact-checking by jointly simulating position semantic reasoning features and knowledge-enhanced entity pair features, while simultaneously learning effective semantic-level and knowledge-level statement representations.

[0005] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0006] The multimodal fact-checking method based on entity augmentation and position checking of this invention is characterized by the following steps:

[0007] Step 1: Collection and preprocessing of multimodal news data;

[0008] Step 1.1: Extract the text content of each multimodal news item on the social media platform and its corresponding image to obtain the news text set. and its corresponding news image collection Among them, T i I represents the i-th news article text; i T represents i The corresponding i-th news image; N represents the total number of multimodal news items;

[0009] Step 1.2: Extract text and image evidence from fact-checking websites for each multimodal news item to obtain a text set of evidence documents. and evidence image set Among them, DT i DI represents the text of the i-th piece of evidence document; i T represents i The corresponding i-th evidence image;

[0010] Step 1.3: Set the authenticity label for the i-th multimodal news item, denoted as y. i , and y i ∈{0,1}; thus constructing the training dataset.

[0011] Step 2: Construct a multimodal fact-checking network, including: a multimodal feature extraction module, a position converter module, and an entity augmentation encoder module;

[0012] Step 3: The multimodal feature extraction module extracts I... i ,T i ,DI i ,DT i Feature extraction is performed to obtain T i global features T C News Text Entity Set I i News Visual Entities Collection News Visual Features I C DT i global features T D Evidence document text entity set DI i Named entity set Visual features of evidence documents I D ;

[0013] Step 4: The position converter module performs I... c I D T D T c Feature fusion is performed to obtain the modal stance features f of the output text. tt Visual modal position features fvv and cross-modal position features f tv ;

[0014] Step 5: Construct the entity augmentation encoder module, including: entity linking unit, knowledge-enhanced distance measurement unit, and symbolic attention feature fusion unit;

[0015] Step 5.1, the entity link unit will Linking to the knowledge graph yields the text modal entity set E. TT Visual modal entity set E VV Cross-modal entity set E TV ;

[0016] Step 5.2: The knowledge-enhanced distance measurement unit calculates E respectively. TT E TT and E TV The knowledge-enhanced distance between each pair of entity embedding features;

[0017] Step 5.3: Use the symbolic attention feature fusion unit to process E respectively. TT E VV E TV After fusion processing, the corresponding text modality knowledge-enhanced entity pairing reasoning feature f is obtained. Ett Visual modal knowledge-enhanced entity pairing reasoning features f Evv Multimodal knowledge-enhanced entity pairing reasoning features f Etv ;

[0018] Step Six: Optimization of the Multimodal Fact-Checking Network:

[0019] Step 6.1: Use equation (9) to predict the probability that the i-th multimodal news item is fake news.

[0020]

[0021] In equation (9), σ represents the sigmoid activation function, W c b represents the weight matrix of the classifier. c Represents the deviation vector;

[0022] Step 6.2: Construct the cross-entropy loss function using equation (10).

[0023]

[0024] Step 6.3: Based on the training dataset X, train the multimodal fake news detection network using the Adam optimization strategy until the network's total loss function is reached. The process continues until convergence, thus obtaining the optimal multimodal fact-checking model, which can be used to fact-check any multimodal news item.

[0025] The multimodal fact-checking method based on entity enhancement and position checking described in this invention is also characterized in that step three includes the following steps:

[0026] Step 3.1: Use the pre-trained BERT model to process the i-th news text T. i Feature extraction is performed to obtain the i-th news text T. i Feature sequence F T,i =[f 1.i ,f 2.i ,...,f l.i ,...,f L.i ], where f l.i This represents the i-th news text T. i The l-th word-level text feature; L represents the number of text features;

[0027] Using a Long Short-Term Memory (LSTM) network to process the feature sequence F T,i Feature extraction is performed, and the hidden state features output by the last step of the Long Short-Term Memory (LSTM) network are used as the i-th news text T. i global features T C ;

[0028] Step 3.2: Use the entity linking tool TAGME to link from T i Extract news text entity sets

[0029] Step 3.3: Use the API tool to identify the i-th news image. i The objects and celebrities in the news form a visual entity set.

[0030] Step 3.4: Utilize the pre-trained model ResNet-50 and the feature pyramid network from I i Extracting visual features from news I C ;

[0031] Step 3.5: Follow the process in steps 3.1-3.4 to process DI. i ,DT i Relevant features are extracted to obtain the i-th piece of evidence text DT. i global features T D Evidence document text entity set Evidence document image DI i Named entity set in Visual features of evidence documents I D .

[0032] Step four includes the following steps:

[0033] Step 4.1, I c I D T D T c Input the shared weight W respectively shared After processing in the MLP layer, the i-th news text T is obtained. i global text feature sequence F Ts,i The i-th piece of evidence document DT i global text features F Gs,i The i-th news image T i Image intermediate features F Vs,i Evidence document image DT i Image intermediate features F Cs,i ;

[0034] Step 4.2: Obtain the news document consistency feature F of the image modality according to equation (1). VV,i F, the consistency feature of news documents in text modality TT,i And cross-modal news text-image consistency description features F TV,i :

[0035]

[0036] In equation (1), Q V,i Indicated by the intermediate features F of news images Vs,i The query vector, K C,i The F represents the intermediate feature sequence based on the evidence text. Cs,i The key vector, V C,i The F represents the intermediate feature sequence based on the evidence text. Cs,i The value vector, d1 represents the dimension of the query vector, key vector, and value vector, softmax represents the mathematical function that maps the real number vector to a probability distribution, T represents the transpose, W VV Q represents the weight matrix of the image modality attention operation. T,i Represents the global feature sequence F based on the news text. Ts,i The query vector, K V,i Indicated by the intermediate features F of news images Vs,i The key vector, V V,i Indicated by the intermediate features F of news images Vs,i value vector, W TV K represents the weight matrix for the news text-image cross-modal attention operation. G,i The global feature sequence F representing the text of the evidence document Gs,i The key vector; VG,i F represents the global feature sequence based on the text of the evidence document. Gs,i value vector, W TT The weight matrix representing the text modality attention operation;

[0037] Step 4.3: Apply self-attention mechanism to F VV,i F TV,i and F TT,i The three modeling features are processed separately, and then further processed using a fully connected layer and an average pooling layer to output the corresponding text modal stance features f. tt Visual modal position features f vv and cross-modal position features f tv .

[0038] Step 5.1 includes the following steps:

[0039] Step 5.1.1: Use the pre-trained entity representation model TransE to represent the news text entity set. and visual entity set Link to the Freebase knowledge graph to obtain text entity embedding features. and visual entity embedding features

[0040] Step 5.1.2: Use the pre-trained entity representation model TransE to analyze the evidence document text entity set. and visual entity set Linking to the Freebase knowledge graph allows us to obtain the embedded features of evidence text entities. and visual entity embedding features

[0041] Step 5.1.3: Embed news text entities into features and evidence document text entity embedding features Composition of text modal entity set E TT Embedding news visual entities into features and visual entity embedding features of evidence documents Composition of visual modal entity set E VV Embedding news text entities into features and news visual entity embedding features Composition of cross-modal entity set E TV .

[0042] Step 5.2 includes the following steps:

[0043] Step 5.2.1, for E TT E VV E TVAny entity embedding feature pair (E) u E v First, find connection E in the Freebase knowledge graph. u and E v Find the shortest path π, and then obtain E according to equations (2) and (3). u Knowledge-enhanced entity representation h u and E v Knowledge-enhanced entity representation h v Thus h u and h v Features after splicing [h] u h v As an entity embedding feature pair (E) u E v A pair of semantically related entity representations:

[0044]

[0045]

[0046] In equations (2) and (3), g j is the entity embedding feature on the shortest path π in the j-th Freebase knowledge graph, M represents the total number of entities on the shortest path π, and α represents the weight coefficient;

[0047] Step 5.2.2: Obtain the knowledge enhancement distance D(E) between a pair of semantically related entity representations according to equation (5). u E v ):

[0048] D(E u E v )=||h u -h v ||2 (4).

[0049] Step 5.3 includes the following steps:

[0050] Step 5.3.1, for E TT Knowledge augmentation distance D(E) u E v The m semantically related entity representations with the smallest knowledge enhancement distance are selected as the consistent entity pair subset E. co,tt and its corresponding distance value D pos,tt We will select the m pairs of semantically related entity representations with the largest knowledge-enhanced distance as the subset E of inconsistent entity pairs. ic,tt and its corresponding distance value D neg,tt ;

[0051] Step 5.3.2: Calculate the consistent entity pair subset E using equations (5) and (6). co,tt Attention score α pos,tt and the corresponding distance value D pos,tt Positive attention characteristics

[0052]

[0053]

[0054] In equation (6), d e For E co,tt The dimension;

[0055] Step 5.3.3: Calculate the inconsistent entity pair subset E using equations (7) and (8). ic,tt negative attention score α neg,tt and the corresponding distance value D neg,tt negative attention characteristics

[0056]

[0057]

[0058] Step 5.3.4, will and The concatenated features are used as a set of intermediate features. After modeling these intermediate features using a self-attention mechanism, fully connected layers and average pooling layers are used to further process the modeled features, ultimately yielding the text modality knowledge-enhanced entity pairing reasoning feature f. Ett ;

[0059] Step 5.3.5: Follow the procedures in steps 5.3.1-5.3.4 for E. VV After processing, the visual modality knowledge-enhanced entity pairing reasoning feature f is finally obtained. Evv ;

[0060] Step 5.3.6: Follow the procedures in steps 5.3.1-5.3.4 for E. TV After processing, the multimodal knowledge-enhanced entity pairing reasoning feature f is finally obtained. Etv .

[0061] The present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor in executing the multimodal fact-checking method, and the processor is configured to execute the program stored in the memory.

[0062] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program is executed by a processor to perform the steps of the multimodal fact-checking method.

[0063] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0064] 1. This invention includes a multimodal feature extraction module, a position changer, and an entity augmentation encoder (EeE). Upon receiving a multimodal claim and its retrieved corresponding evidence, the multimodal feature extraction module first extracts valuable clues from text, images, and entities, thereby providing diverse clues for fact-checking and facilitating the discovery of news features.

[0065] 2. This invention utilizes three position transformers to simulate the hierarchical reasoning process of humans, checking the consistency between different types of claims and evidence. The position transformers first introduce a set of shared prototypes as queries, guiding the reconstruction of the feature representations of claims and corresponding evidence, thereby projecting the two types of features into a common feature space to reduce computational load. Then, a fusion layer is further used to obtain semantic reasoning features from the three positions, improving the accuracy and efficiency of multimodal fact-checking from the perspective of simulating human reasoning.

[0066] 3. The Entity Augmentation Encoder (EeE) included in this invention constructs a cross-modal entity pair set and two unimodal entity pair sets within a multimedia post, and designs a knowledge-related reasoning strategy to find the shortest semantically relevant path between each pair of entities in an external knowledge graph. By absorbing supplementary background knowledge of all entities in this path, the knowledge augmentation distance and entity representation are refined at a higher knowledge level. Then, it selects the closest / farthest entity pair as the most consistent / least inconsistent pair, and further fuses the selected entity pairs by employing a symbolic attention mechanism to capture the features of consistent and inconsistent knowledge-augmented entity pairs, thereby introducing external knowledge and improving the accuracy and efficiency of multimodal fact-checking. Attached Figure Description

[0067] Figure 1 This is a schematic diagram of the overall structure of the present invention;

[0068] Figure 2 This is a structural diagram of a position converter;

[0069] Figure 3 This is a structural diagram of a solid augmented encoder. Detailed Implementation

[0070] In this embodiment, a multimodal fact-checking method based on entity augmentation and position checking (ESCNet) includes: a multimodal feature extraction module, a position transformer, and an entity augmentation encoder. ESCNet jointly models position semantic reasoning features and knowledge-enhanced entity pair features to simultaneously learn effective semantic-level and knowledge-level assertion news representations, thereby enabling more accurate multimodal fact checking. Specifically, the method's process is as follows... Figure 1 Specifically, it is done in the following steps:

[0071] Step 1: Collection and preprocessing of multimodal news data;

[0072] Step 1.1: Extract the text content of each multimodal news item on the social media platform and its corresponding image to obtain the news text set. and its corresponding news image collection Among them, T i I represents the i-th news article text; i T represents i The corresponding i-th news image; N represents the total number of multimodal news items;

[0073] Step 1.2: Extract text and image evidence from fact-checking websites for each multimodal news item to obtain a text set of evidence documents. and evidence image set Among them, DT i DI represents the text of the i-th piece of evidence document; i T represents i The corresponding i-th evidence image;

[0074] Step 1.3: Set the authenticity label for the i-th multimodal news item, denoted as y. i , and y i ∈{0,1}; thus constructing the training dataset.

[0075] Step 2: Construct a multimodal fact-checking network, including: a multimodal feature extraction module, a position converter module, and an entity augmentation encoder module.

[0076] Step 3: The multimodal feature extraction module extracts I... i ,T i ,DI i ,DT i Feature extraction is performed to obtain T i global features T C News Text Entity Set I i News Visual Entities Collection News Visual Features I C DT iglobal features T D Evidence document text entity set DI i Named entity set Visual features of evidence documents I D ;

[0077] Step 3.1: Use the pre-trained BERT model to process the i-th news text T. i Feature extraction is performed to obtain the i-th news text T. i Feature sequence F T,i =[f 1.i ,f 2.i ,...,f l.i ,...,f L.i ], where f l.i T represents the i-th news text. i The l-th word-level text feature; L represents the number of text features;

[0078] Using a Long Short-Term Memory (LSTM) network to process the feature sequence F T,i Feature extraction is performed, and the hidden state features output by the last step of the Long Short-Term Memory (LSTM) network are used as the i-th news text T. i global features T C .

[0079] Step 3.2: Use the entity linking tool TAGME to link from T i Extract news text entity sets In this embodiment, the number of entities to be extracted is set to 5;

[0080] Step 3.3: Use the API tool to identify the i-th news image. i The objects and celebrities in the news form a visual entity set. In this embodiment, the number of entities extracted is set to 5.

[0081] Step 3.4: Utilize the pre-trained model ResNet-50 and the feature pyramid network from I i Extracting visual features from news I C ;

[0082] Step 3.5: Follow the process in steps 3.1-3.4 to process DI. i ,DT i Relevant features are extracted to obtain the i-th piece of evidence text DI. i global features T D Evidence document text entity set Evidence document image DI i Named entity set in Visual features of evidence documents ID .

[0083] Step 4: The position converter module performs I... c I D T D T c Feature fusion is performed to obtain the modal stance features f of the output text. tt Visual modal position features f vv and cross-modal position features f tv ;

[0084] Step 4.1: Construct an MLP layer consisting of one m×n linear layer and one ReLU activation function layer, and then... c I D T D T c Input the shared weight W respectively shared After processing in the MLP layer, the i-th news text T is obtained. i global text feature sequence F Ts,i The i-th piece of evidence document DT i global text features F Gs,i The i-th news image T i Image intermediate features F Vs,i Evidence document image DT i Image intermediate features F Cs,i .

[0085] Step 4.2: Obtain the news document consistency feature F of the image modality according to equation (1). VV,i F, the consistency feature of news documents in text modality TT,i And cross-modal news text-image consistency description features F TV,i :

[0086]

[0087] In equation (1), Q V,i Indicated by the intermediate features F of news images Vs,i The query vector, K C,i The F represents the intermediate feature sequence based on the evidence text. Cs,i The key vector, V C,i The F represents the intermediate feature sequence based on the evidence text. Cs,i The value vector, d1 represents the dimension of the query vector, key vector, and value vector. In this embodiment, the size of d1 is set to 256. softmax represents a mathematical function that maps real number vectors to probability distributions. T represents transpose. W VV Q represents the weight matrix of the image modality attention operation. T,i Represents the global feature sequence F based on the news text.Ts,i The query vector, K V,i Indicated by the intermediate features F of news images Vs,i The key vector, V V,i Indicated by the intermediate features F of news images Vs,i value vector, W TV K represents the weight matrix for the news text-image cross-modal attention operation. G,i The global feature sequence F representing the text of the evidence document Gs,i The key vector; V G,i F represents the global feature sequence based on the text of the evidence document. Gs,i value vector, W TT This represents the weight matrix for the text modality attention operation.

[0088] Step 4.3: Apply self-attention mechanism to F VV,i F TV,i and F TT,i The three modeling features are processed separately, and then further processed using a fully connected layer and an average pooling layer to output the corresponding text modal stance features f. tt Visual modal position features f vv and cross-modal position features f tv .

[0089] Step 5: Construct the entity augmentation encoder module, including: entity linking unit, knowledge-enhanced distance measurement unit, and symbolic attention feature fusion unit;

[0090] Step 5.1, the entity link unit will Linking to the knowledge graph yields the text modal entity set E. TT Visual modal entity set E VV Cross-modal entity set E TV .

[0091] Step 5.1.1: Use the pre-trained entity representation model TransE to represent the news text entity set. and visual entity set Link to the Freebase knowledge graph to obtain text entity embedding features. and visual entity embedding features

[0092] Step 5.1.2: Use the pre-trained entity representation model TransE to analyze the evidence document text entity set. and visual entity set Linking to the Freebase knowledge graph allows us to obtain the embedded features of evidence text entities. and visual entity embedding features

[0093] Step 5.1.3: Embed news text entities into features and evidence document text entity embedding features Composition of text modal entity set E TT Embedding news visual entities into features and visual entity embedding features of evidence documents Composition of visual modal entity set E VV Embedding news text entities into features and news visual entity embedding features Composition of cross-modal entity set E TV .

[0094] Step 5.2: The knowledge-enhanced distance measurement unit calculates E respectively. TT E TT and E TV The knowledge-enhanced distance between each pair of entity embedding features;

[0095] Step 5.2.1, for E TT E VV F TV Any entity embedding feature pair (E) u E v First, find connection E in the Freebase knowledge graph. u and E v Find the shortest path π, and then obtain E according to equations (2) and (3). u Knowledge-enhanced entity representation h u and E v Knowledge-enhanced entity representation h v Thus h u and h v Features after splicing [h] u h v As an entity embedding feature pair (E) u E v A pair of semantically related entity representations:

[0096]

[0097]

[0098] In equations (2) and (3), g j It is the entity embedding feature on the shortest path π in the j-th Freebase knowledge graph, where M represents the total number of entities on the shortest path π, and α represents the weight coefficient; in this embodiment, α, α j α M-j The size is set to 0.9.

[0099] Step 5.2.2: Obtain the knowledge enhancement distance D(E) between a pair of semantically related entity representations according to equation (5). u E v ):

[0100] D(E u E v )=||h u -h v ||2 (4) Step 5.3: Use the symbolic attention feature fusion unit to process E respectively TT E VV E TV After fusion processing, the corresponding text modality knowledge-enhanced entity pairing reasoning feature f is obtained. Ett Visual modal knowledge-enhanced entity pairing reasoning features f Evv Multimodal knowledge-enhanced entity pairing reasoning features f Etv .

[0101] Step 5.3.1, for E TT Knowledge augmentation distance D(E) u E v The m semantically related entity representations with the smallest knowledge enhancement distance are selected as the consistent entity pair subset E. co,tt and its corresponding distance value D pos,tt We will select the m pairs of semantically related entity representations with the largest knowledge-enhanced distance as the subset E of inconsistent entity pairs. ic,tt and its corresponding distance value D net,tt ;

[0102] Step 5.3.2: For the consistent entity pair subset E co,tt and its corresponding distance value D pos,tt The corresponding attention score α is calculated using equations (5) and (6). pos,tt and positive attention characteristics

[0103]

[0104]

[0105] In equation (6), d e For E co,tt The dimension; in this embodiment, d e The size is set to 256.

[0106] Step 5.3.3: For the inconsistent entity pair subset E ic,tt and its corresponding distance value D neg,tt The corresponding negative attention score α is calculated using equations (7) and (8). neg,tt and negative attention characteristics

[0107]

[0108]

[0109] In equation (6), d e For E ic,tt The dimension; in this embodiment, d e The size is set to 256.

[0110] Step 5.3.4, will and After concatenation, these intermediate features are used as a set of intermediate features. A self-attention mechanism is applied to model these intermediate features, and then fully connected layers and average pooling layers are used to further process the modeled features, ultimately yielding the text modality knowledge-enhanced entity pairing reasoning feature f. Ett ;

[0111] Step 5.3.5: Follow the procedures in steps 5.3.1-5.3.4 for E. VV After processing, the visual modality knowledge-enhanced entity pairing reasoning feature f is finally obtained. Evv ;

[0112] Step 5.3.6: Follow the procedures in steps 5.3.1-5.3.4 for E. TV After processing, the multimodal knowledge-enhanced entity pairing reasoning feature f is finally obtained. Etv .

[0113] Step Six: Optimization of the Multimodal Fact-Checking Network:

[0114] Step 6.1: Use equation (9) to predict the probability that the i-th multimodal news item is fake news.

[0115]

[0116] In equation (9), σ represents the sigmoid activation function, W c b represents the weight matrix of the classifier. c This represents the deviation vector.

[0117] Step 6.2: Construct the cross-entropy loss function using equation (10).

[0118]

[0119] Step 6.3: Based on the training dataset X, train the multimodal fake news detection network using the Adam optimization strategy until the network's total loss function is reached. The process continues until convergence, thus obtaining the optimal multimodal fact-checking model, which can be used to fact-check any multimodal news item.

[0120] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0121] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

Claims

1. A multimodal fact-checking method based on entity augmentation and position checking, characterized in that, Includes the following steps: Step 1: Collection and preprocessing of multimodal news data; Step 1.1: Extract the text content of each multimodal news item on the social media platform and its corresponding image to obtain the news text set. and its corresponding news image collection ;in, Indicates the first i News text; express The corresponding number i Zhang News Image; N This represents the total number of multimodal news items. Step 1.2: Extract text and image evidence from fact-checking websites for each multimodal news item to obtain a text set of evidence documents. and evidence image set ;in, Indicates the first i One piece of evidence document text; express The corresponding number i Zhang's evidence images; Step 1.3, Set the first i The authenticity label for multimodal news is denoted as... ,and Thus, a training dataset is constructed. ; Step 2: Construct a multimodal fact-checking network, including: a multimodal feature extraction module, a position converter module, and an entity augmentation encoder module; Step 3: The multimodal feature extraction module... Feature extraction is performed to obtain global features News Text Entity Set , News Visual Entities Collection News visual features , global features Evidence document text entity set , Named entity set Visual features of evidence documents ; Step 4: Position converter module , , , Feature fusion is performed to obtain the modal stance features of the output text. Visual modal position characteristics and cross-modal position features ; Step 5: Construct the entity augmentation encoder module, including: entity linking unit, knowledge-enhanced distance measurement unit, and symbolic attention feature fusion unit; Step 5.1, the entity link unit will , , , Linking to the knowledge graph yields a text modal entity set. Visual modal entity set Cross-modal entity sets ; Step 5.2: The knowledge-enhanced distance measurement unit calculates respectively. , and The knowledge-enhanced distance between each pair of entity embedding features; Step 5.2.1, for , , Any entity embedding feature pair First of all Freebase Finding connections in a knowledge graph and shortest path Then, according to equations (2) and (3), we obtain Knowledge-enhanced entity representation and Knowledge-enhanced entity representation Thus and Features after splicing As entity embedding feature pair A pair of semantically related entity representations: (2) (3) In equations (2) and (3), It is the first j indivual Freebase Shortest path in knowledge graph Entity embedding features on Represents the shortest path The total number of entities on Indicates the weighting coefficient; Step 5.2.2: Obtain the knowledge enhancement distance of a pair of semantically related entity representations according to equation (5). : (4) Step 5.3: Use the symbolic attention feature fusion unit to respectively... , , After fusion processing, the corresponding text modality knowledge-enhanced entity pairing reasoning features are obtained. Visual modal knowledge-enhanced entity pairing reasoning features Multimodal knowledge-enhanced entity pairing reasoning features ; Step 5.3.1, targeting Knowledge Enhancement Distance Select the one with the smallest knowledge augmentation distance The semantically related entity representations are subsets of consistent entity pairs. and their corresponding distance values The one with the largest knowledge augmentation distance will be selected. The semantically related entity representations are subsets of inconsistent entity pairs. and their corresponding distance values ; Step 5.3.2: Calculate the consistent entity pair subset using equations (5) and (6). Attention score and the corresponding distance value Positive attention characteristics : (5) (6) In equation (6), for The dimension; Step 5.3.3: Calculate the subset of inconsistent entity pairs using equations (7) and (8). negative attention score and the corresponding distance value negative attention characteristics : (7) (8) Step 5.3.4, will and The concatenated features are used as a set of intermediate features. After modeling these intermediate features using a self-attention mechanism, fully connected layers and average pooling layers are used to further process the modeled features, ultimately yielding text modality knowledge-enhanced entity pairing reasoning features. ; Step 5.3.5: Follow the process in steps 5.3.1-5.3.

4. After processing, the final result is the visual modality knowledge-enhanced entity pairing reasoning feature. ; Step 5.3.6: Follow the procedures in steps 5.3.1-5.3.

4. After processing, the multimodal knowledge-enhanced entity pairing reasoning features are finally obtained. ; Step Six: Optimization of the Multimodal Fact-Checking Network: Step 6.1: Use equation (9) to predict the first... i The probability that a multimodal news item is fake news. ; (9) In equation (8), represent sigmoid Activation function This represents the weight matrix of the classifier. Represents the deviation vector; Step 6.2: Construct the cross-entropy loss function using equation (10). : (10) Step 6.3: Based on the training dataset ,use Adam The optimization strategy is used to train the multimodal fake news detection network until the network's total loss function is reached. The process continues until convergence, thus obtaining the optimal multimodal fact-checking model, which can be used to fact-check any multimodal news item.

2. The multimodal fact-checking method based on entity augmentation and position checking according to claim 1, characterized in that, Step three includes the following steps: Step 3.1: Use pre-trained... BERT Model for the first i News text Feature extraction is performed to obtain the first... i News text Feature sequences ,in, Indicates the first i News text The Middle l Text features at the word level; L Indicates the number of text features; Using Long Short-Term Memory Networks LSTM For feature sequences Feature extraction is performed, and the Long Short-Term Memory network is used. LSTM The hidden state feature of the last step output is used as the first i News text global features ; Step 3.2: Utilize entity linking tools TAGM E from Extract news text entity sets ; Step 3.3, Use API Tool recognition of the first i Zhang News Image The objects and celebrities in the news form a visual entity set. ; Step 3.4: Utilize the pre-trained model ResNet-50 and the feature pyramid network from... Extracting visual features from news ; Step 3.5: Follow the process in steps 3.1-3.

4. Relevant features were extracted separately to obtain the first... i Evidence text global features Evidence document text entity set Evidence documents and images Named entity set in Visual features of evidence documents .

3. The multimodal fact-checking method based on entity augmentation and position checking according to claim 2, characterized in that, Step four includes the following steps: Step 4.1, , , , Input the shared weights respectively of MLP After processing in the layer, the first... i News text Global text feature sequence , No. i Evidence document Global text features , No. i Zhang News Image Image intermediate features Evidence document images Image intermediate features ; Step 4.2: Obtain the news document consistency features of the image modality according to equation (1). Consistency features of news documents in text modality Cross-modal news text-image consistency description features : (1) In equation (1), Indicated based on intermediate features of news images The query vector, Indicates the intermediate feature sequence based on evidence text The key vector, Indicates the intermediate feature sequence based on evidence text The value vector, This indicates the dimensions of the query vector, key vector, and value vector. This represents a mathematical function that maps a real number vector to a probability distribution. T Indicates transpose. The weight matrix represents the image modality attention operation. Represents global feature sequences based on news text The query vector, Indicated based on intermediate features of news images The key vector, Indicated based on intermediate features of news images The value vector, This represents the weight matrix for the cross-modal attention operation between news text and images. Global feature sequence representing the text of evidence documents The key vector; Represents a global feature sequence based on evidence document text. The value vector, The weight matrix representing the text modality attention operation; Step 4.3: Apply self-attention mechanism to and The three modeling features are processed separately, and then further processed using a fully connected layer and an average pooling layer to output the corresponding text modality stance features. Visual modal position characteristics and cross-modal position features .

4. The multimodal fact-checking method based on entity augmentation and position checking according to claim 3, characterized in that, Step 5.1 includes the following steps: Step 5.1.1: Use the pre-trained entity representation model TransE News text entity set and visual entity set Link to Freebase In knowledge graphs, text entity embedding features are obtained. and visual entity embedding features ; Step 5.1.2: Use the pre-trained entity representation model TransE Evidence document text entity set and visual entity set Link to Freebase In knowledge graphs, thus obtaining evidence text entity embedding features. and visual entity embedding features ; Step 5.1.3: Embed news text entities into features and evidence document text entity embedding features Composition of text modal entity sets Embedding news visual entities into features and visual entity embedding features of evidence documents Composition of visual modal entity set Embedding news text entities into features and news visual entity embedding features Composition of cross-modal entity sets .

5. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the multimodal fact-checking method of any one of claims 1-4, the processor being configured to execute the program stored in the memory.

6. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is run by the processor, it performs the steps of the multimodal fact-checking method according to any one of claims 1-4.

Images