Multimodal rumor detection method based on double evidence enhancement and image-text similarity perception

Abstract

The present invention proposes a multimodal rumor detection method based on dual evidence enhancement and image-text similarity perception, the method comprising: using a pre-trained language model and a pre-trained visual model to extract text semantic features and image visual features of the post to be detected respectively; using ELA to mine potential tampering traces in the image; obtaining relevant news reports through reverse search of the post image, searching and screening relevant images using the post text, and realizing interactive alignment of evidence with the post to be verified through a cross-attention mechanism; using the EGTR algorithm and the Flan‑T5 model to convert images and texts into visual scene graphs and text scene graphs respectively; extracting relevant background knowledge through knowledge distillation; and learning the scene graph through an improved graph attention network. The present invention designs a feature hierarchical fusion method based on a gated neural network to achieve adaptive fusion of features, and inputs the fused features into a multi-layer perceptron for result classification.

Description

Technical Field

[0001] The present invention relates to the fields of computer science and artificial intelligence technology, and in particular to a multimodal rumor detection method based on double evidence enhancement and image-text similarity perception. Background Art

[0002] With the popularity of social media and online platforms, the speed of information dissemination has greatly increased. However, the spread of false information and online rumors has also become a serious problem. Traditional rumor detection methods mostly focus on the analysis of text content, classifying and identifying text through natural language processing technology. Although such methods can identify rumors to a certain extent, with the diversification of rumor propagation methods, especially the widespread use of images and videos, the method of relying solely on text analysis can no longer meet the needs of rumor detection in complex scenarios. False information is often accompanied by image or video content, which may be tampered with or misleading. Simple text analysis cannot effectively identify the potential deception.

[0003] In addition, the determination of online rumors usually requires the support of external knowledge and multiple evidences. Although existing rumor detection methods can process text and image information, they often ignore the background information extracted from external knowledge bases. When faced with deep semantic deviations, fuzzy information, and contextual ambiguity, the lack of effective use of these external evidences limits the detection effect of existing methods. Therefore, how to integrate external knowledge with multimodal information and enhance the semantic understanding of text and image content has become the key to improving the accuracy and robustness of rumor detection.

[0004] Although deep learning has made many advances in multimodal tasks of images and texts, existing technical methods still face many challenges in the field of rumor detection. Existing image-text fusion models often ignore the semantic association between images and texts, resulting in the inability to fully identify the deep-level features of rumors. In addition, existing technologies lack an adaptive mechanism for the fusion of multimodal information, making it difficult to effectively handle the changes and noise of different modal features at different time points. Summary of the invention

[0005] In view of the above situation, the main purpose of the present invention is to propose a multimodal rumor detection method based on double evidence enhancement and image-text similarity perception to solve the above technical problems.

[0006] The present invention proposes a multimodal rumor detection method based on double evidence enhancement and image-text similarity perception, and the method comprises the following steps:

[0007] Step 1: Use the pre-trained language model combined with the bidirectional long short-term memory network to extract features from the post to obtain the semantic features of the post text; use the pre-trained visual model to extract features from the post image to obtain the image visual features of the post image;

[0008] Step 2: Perform difference analysis on the post image using the error level analysis method to obtain a difference analysis image, and extract tampering features from the difference analysis image using the pre-trained visual model to obtain image tampering features;

[0009] Step 3: reverse search the post images to obtain target news reports, and filter the target news reports to obtain text filtering evidence, use the post text to search for images to obtain target image information, and filter the target image information to obtain image filtering evidence;

[0010] Step 4: Interactively align the text screening evidence and the image screening evidence through a multi-head cross attention mechanism to obtain aligned text evidence features and aligned image evidence features, and verify the consistency between the aligned text evidence features and the aligned image evidence features and the input post;

[0011] Step 5: Use the EGTR algorithm to calculate the post image to generate a visual scene graph, and use the Flan-T5 model to process the post text to generate a text scene graph;

[0012] Step 6: Combine the visual scene graph and the text scene graph with knowledge distillation in sequence to obtain target background knowledge, and integrate the target background knowledge into the visual scene graph and the text scene graph to obtain a knowledge-enhanced visual scene graph and a knowledge-enhanced text scene graph;

[0013] Step 7: Use the graph attention network to perform graph-text semantic matching on the knowledge-enhanced visual scene graph and the knowledge-enhanced text scene graph to obtain the graph-text semantic matching result of the visual scene graph and the text scene graph;

[0014] Step 8: Adaptively perform hierarchical fusion on the semantic features of the text, the aligned text evidence features, the image visual features, the aligned picture evidence features, the image tampering features, and the image-text semantic matching features using a feature hierarchical fusion algorithm to obtain a hierarchical final fusion result, wherein the feature hierarchical fusion algorithm is constructed based on a gated neural network;

[0015] Step 9: Input the final fusion result of the layers into a multi-layer perceptron to obtain the result classification.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] 1. This invention overcomes the problem of insufficient image-text fusion in traditional methods by fusing image and text features and combining the guidance of external knowledge graphs, and enhances the understanding of deep semantic relationships and complex background information;

[0018] 2. The present invention adopts error level analysis technology and cross attention mechanism to improve the accuracy of image-text matching.

[0019] 3. The present invention realizes the adaptive fusion of multimodal features through a feature hierarchical fusion method based on a gated neural network, thereby improving the accuracy of rumor detection and the adaptability of the model;

[0020] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flowchart of the steps of the multimodal rumor detection method based on double evidence enhancement and image-text similarity perception proposed in the present invention.

[0022] Figure 2 Schematic diagram of the visual model architecture of the multimodal rumor detection method based on double evidence enhancement and image-text similarity perception proposed in the present invention.

[0023] Figure 3 This is a multi-head cross-attention calculation flowchart of the multimodal rumor detection method based on double evidence enhancement and image-text similarity perception proposed in the present invention.

[0024] Figure 4 Schematic diagram of the overall architecture of the EGTR of the multimodal rumor detection method based on double evidence enhancement and image-text similarity perception proposed in the present invention.

[0025] Figure 5 Schematic diagram of knowledge distillation of the multimodal rumor detection method based on double evidence enhancement and image-text similarity perception proposed in the present invention.

[0026] Figure 6 This is a diagram of the architecture of a multimodal rumor detection model of the multimodal rumor detection method based on double evidence enhancement and image-text similarity perception proposed in the present invention. DETAILED DESCRIPTION

[0027] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout are the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0028] These and other aspects of the embodiments of the present invention will be apparent with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to provide some ways to implement the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0029] See also Figure 1 The embodiment of the present invention proposes a multimodal rumor detection method based on double evidence enhancement and image-text similarity perception, and the method comprises the following steps:

[0030] Step 1: Use the pre-trained language model combined with the bidirectional long short-term memory network to extract features from the post to obtain the semantic features of the post text; use the pre-trained visual model to extract features from the post image to obtain the image visual features of the post image;

[0031] In step 1, in the step of extracting features from posts to obtain semantic features of post texts by combining a pre-trained language model with a bidirectional long short-term memory network, the relationship corresponding to the text semantic feature calculation process is:

[0032] ;

[0033] in, Represents the semantic features of text, Represents the input text sequence, represents the semantic temporal feature extraction operation, Represents the text semantic feature extraction operation.

[0034] See also Figure 2 In the step of extracting features from the post image using the pre-trained visual model to obtain the image visual features of the post image, the corresponding relationship between the calculation process of the image visual features is:

[0035] ;

[0036] in, Represents the visual features of the image, represents the visual feature extraction operation of the image, Represents the input image.

[0037] Step 2: Use the error level analysis method to perform difference analysis on the post image to obtain a difference analysis image, and use the pre-trained visual model to extract tampering features on the difference analysis image to obtain image tampering features.

[0038] In step 2, in the step of performing difference analysis on the post image using the error level analysis method to obtain a difference analysis image, and extracting tampering features from the difference analysis image using the pre-trained visual model to obtain image tampering features, the calculation process of the image tampering features corresponds to the relationship between:

[0039] ;

[0040] in, represents the ELA image tampering feature, represents the input ELA image, Indicates taking the average value of all block features.

[0041] Step 3: Perform a reverse search on the post images to obtain target news reports, and filter the target news reports to obtain text filtering evidence, use the post text to search for images to obtain target image information, and filter the target image information to obtain image filtering evidence.

[0042] Step 4: Interactively align the text screening evidence and the image screening evidence through a multi-head cross-attention mechanism to obtain aligned text evidence features and aligned image evidence features, and verify the consistency between the aligned text evidence features and the aligned image evidence features and the input post.

[0043] See also Figure 3 In step 4, in the step of interactively aligning the text screening evidence and the image screening evidence through a multi-head cross attention mechanism to obtain aligned text evidence features and aligned image evidence features, and verifying the consistency between the aligned text evidence features and the aligned image evidence features and the input post, the calculation process of verifying the consistency between the text screening evidence and the image screening evidence and the input post corresponds to the relationship between:

[0044] ;

[0045] in, represents the input sequence, Representation and Different input sequences, represents the operation of multi-head cross attention mechanism, Represents a splicing operation, represents the first attention head, Indicates Attention head, represents the parameter matrix that needs to be learned, Indicates Attention head, represents the cross attention mechanism operation, Indicates a query, Indicates the key, Indicates the value, Indicates The weight matrix of the attention head, Indicates The weight matrix of the attention head, Indicates The weight matrix of the attention head, represents the normalization operation, Represents the key-value dimension, represents the transpose key, Represents the transpose of a matrix.

[0046] Step 5: Use the EGTR algorithm to calculate the post image to generate a visual scene graph, and use the Flan-T5 model to process the post text to generate a text scene graph.

[0047] For further information, see Figure 4 , EGTR (a lightweight scene graph generation method based on Transformer) is used to calculate the image of the post to generate a visual scene graph; the Flan-T5 model fine-tuned on the FACTUAL dataset is used to process the text of the post to generate a text scene graph.

[0048] Step 6: Combine the visual scene graph and the text scene graph with knowledge distillation in turn to obtain the target background knowledge, and integrate the target background knowledge into the visual scene graph and the text scene graph to obtain a knowledge-enhanced visual scene graph and a knowledge-enhanced text scene graph.

[0049] For further information, see Figure 5 Through knowledge distillation, conceptual knowledge related to visual scene graph and text scene graph entities is extracted from external large-scale knowledge graphs to enhance the model's semantic understanding ability of visual scene graphs and text scene graphs.

[0050] Step 7: Use the graph attention network to perform graph-text semantic matching on the knowledge-enhanced visual scene graph and the knowledge-enhanced text scene graph to obtain the graph-text semantic matching results of the visual scene graph and the text scene graph.

[0051] See also Figure 6 In step 7, the graph attention network is used to perform graph-text semantic matching on the knowledge-enhanced visual scene graph and the knowledge-enhanced text scene graph to obtain the graph-text semantic matching result of the visual scene graph and the text scene graph, which specifically includes the following sub-steps:

[0052] The knowledge-enhanced visual scene graph and the text scene graph are sequentially input into the graph attention network, and the relationship features are introduced for mechanism processing to learn the knowledge-enhanced visual scene graph and the text scene graph to obtain the learned visual scene graph and the learned text scene graph;

[0053] The feature vectors of all nodes in the learned visual scene graph and the learned text scene graph are processed by the average pooling mechanism in turn to obtain the whole-image features of the visual scene graph and the whole-image features of the text scene graph respectively;

[0054] The whole-image features of the visual scene graph and the whole-image features of the text scene graph are concatenated and input into a fully connected neural network to obtain the image-text semantic matching results of the visual scene graph and the text scene graph.

[0055] In the step of sequentially inputting the knowledge-enhanced visual scene graph and the text scene graph into the graph attention network, and introducing the relational features for mechanism processing to learn the knowledge-enhanced visual scene graph and the text scene graph to obtain the learned visual scene graph and the learned text scene graph, the corresponding relationship between the calculation process of the learned visual scene graph and the learned text scene graph is:

[0056] ;

[0057] in, represents the graph attention mechanism operation, represents the updated features, Represents a splicing operation, represents the number of multi-head attention, Representation Node The set of neighboring nodes of Indicated in Adjacent nodes in attention type For Node The attention weight, The embedding representation of the node is the learning parameter matrix, The embedding representation of the edge is the learning parameter matrix, represents the residual structure parameter matrix, Representation Node The characteristic representation of Representation Node The characteristic representation of Represents edge The characteristic representation of represents the normalization operation, represents the activation function, represents the attention weight.

[0058] In the step of splicing the whole image features of the visual scene graph with the whole image features of the text scene graph and inputting them into the fully connected neural network to obtain the image-text semantic matching results of the visual scene graph and the text scene graph, the calculation process of the image-text semantic matching results of the visual scene graph and the text scene graph corresponds to the relationship between:

[0059] ;

[0060] in, Represents the final whole image features, represents the visual scene graph, represents the initial features of the node, represents the initial features of the edge, Represents the final whole image features, Represents a text scene graph, Represents the semantic matching features of images and texts, represents the activation function, represents the parameter matrix that can be learned.

[0061] Furthermore, by integrating the relational features of the graph attention network, relational features are introduced when calculating the attention weights and feature aggregation, the knowledge-enhanced visual scene graph and text scene graph are learned, and then the feature vectors of all nodes in the visual scene graph features and the text scene graph are average pooled to obtain the final whole-graph features of the visual scene graph and the final whole-graph features of the text scene graph, where the initial features of the visual scene graph features and the text scene graph nodes and the initial features of the edges are the semantic vectors extracted by the pre-trained language model. Finally, the visual scene graph features and the text scene graph features are concatenated and input into a fully connected neural network to learn their semantic consistency features and obtain the image-text semantic matching results of the visual scene graph and the text scene graph.

[0062] Step 8: Use a feature hierarchical fusion algorithm to adaptively fuse text semantic features, aligned text evidence features, image visual features, aligned picture evidence features, image tampering features, and image-text semantic matching features to obtain a hierarchical final fusion result, wherein the feature hierarchical fusion algorithm is constructed based on a gated neural network.

[0063] In step 8, the feature hierarchical fusion algorithm is used to adaptively fuse the semantic features of the text, the aligned text evidence features, the image visual features, the aligned picture evidence features, the image tampering features and the image-text semantic matching features to obtain the hierarchical final fusion result, which specifically includes the following sub-steps:

[0064] The feature hierarchical fusion algorithm is used to adaptively fuse text semantic features with aligned text evidence features to obtain the first-layer feature fusion result;

[0065] Among them, the calculation process of the first layer feature fusion result corresponds to the following relationship:

[0066] ;

[0067] ;

[0068] in, represents the fusion weight, represents the activation function, Represents a splicing operation, Represents the semantic features of text, Characteristic of textual evidence representing alignment, represents the second parameter matrix to be learned, Represents the first layer feature fusion result;

[0069] The feature hierarchical fusion algorithm is used to adaptively fuse the image visual features with the aligned picture evidence features to obtain the second-layer feature fusion result;

[0070] Among them, the corresponding relationship between the calculation process of the second-layer feature fusion result is as follows:

[0071] ;

[0072] in, represents the third parameter matrix to be learned, Represents the visual features of the image, Represents the aligned image evidence features, Represents the second layer feature fusion result;

[0073] The feature layered fusion algorithm is used to adaptively fuse the image tampering features with the image-text semantic matching features to obtain the third-layer feature fusion result;

[0074] Among them, the calculation process of the third-layer feature fusion result corresponds to the following relationship:

[0075] ;

[0076] in, represents the third-layer feature fusion result, represents the fourth parameter matrix to be learned, Represents the semantic matching features of images and texts, Indicates the characteristics of image tampering;

[0077] The feature layered fusion algorithm is used to adaptively fuse the first layer feature fusion result with the second layer feature fusion result and the third layer feature fusion result to obtain the final fusion result;

[0078] Among them, the calculation process of the third-layer feature fusion result corresponds to the following relationship:

[0079] ;

[0080] in, represents the final fusion result, represents the fifth parameter matrix to be learned, represents the sixth parameter matrix to be learned, represents the seventh parameter matrix to be learned, represents the second fusion weight, represents the third fusion weight, Represents the fourth fusion weight.

[0081] Step 9: Input the final fusion result of the layers into a multi-layer perceptron to obtain the result classification.

[0082] In the step of inputting the final fusion result of the layers into the multilayer perceptron to obtain the result classification, the corresponding relationship in the calculation process of the result classification is:

[0083] ;

[0084] in, Represents the predicted output result, represents the parameters of the linear layer, represents the final fusion result, represents the bias top of the linear layer, Represents the operation of the multi-layer perceptron mechanism.

[0085] Furthermore, the final fusion result of the layers is input into a multi-layer perceptron, and finally the classification result is output through a normalization mechanism operation.

[0086] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0087] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0088] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A multimodal rumor detection method based on double evidence enhancement and image-text similarity perception, characterized in that: The method comprises the following steps: Step 1: Use the pre-trained language model combined with the bidirectional long short-term memory network to extract features from the post to obtain the semantic features of the post text; use the pre-trained visual model to extract features from the post image to obtain the image visual features of the post image; Step 2: Perform difference analysis on the post image using the error level analysis method to obtain a difference analysis image, and extract tampering features from the difference analysis image using the pre-trained visual model to obtain image tampering features; Step 3: reverse search the post images to obtain target news reports, and filter the target news reports to obtain text filtering evidence, use the post text to search for images to obtain target image information, and filter the target image information to obtain image filtering evidence; Step 4: Interactively align the text screening evidence and the image screening evidence through a multi-head cross attention mechanism to obtain aligned text evidence features and aligned image evidence features, and verify the consistency between the aligned text evidence features and the aligned image evidence features and the input post; Step 5: Use the EGTR algorithm to calculate the post image to generate a visual scene graph, and use the Flan-T5 model to process the post text to generate a text scene graph; Step 6: Combine the visual scene graph and the text scene graph with knowledge distillation in sequence to obtain target background knowledge, and integrate the target background knowledge into the visual scene graph and the text scene graph to obtain a knowledge-enhanced visual scene graph and a knowledge-enhanced text scene graph; Step 7: Use the graph attention network to perform graph-text semantic matching on the knowledge-enhanced visual scene graph and the knowledge-enhanced text scene graph to obtain the graph-text semantic matching result of the visual scene graph and the text scene graph; Step 8: Adaptively perform hierarchical fusion on the semantic features of the text, the aligned text evidence features, the image visual features, the aligned picture evidence features, the image tampering features, and the image-text semantic matching features using a feature hierarchical fusion algorithm to obtain a hierarchical final fusion result, wherein the feature hierarchical fusion algorithm is constructed based on a gated neural network; Step 9: Input the final fusion result of the layers into a multi-layer perceptron to obtain the result classification.

2. According to claim 1, the multimodal rumor detection method based on dual evidence enhancement and image-text similarity perception is characterized in that: In step 1, in the step of extracting features from posts to obtain semantic features of post texts by combining a pre-trained language model with a bidirectional long short-term memory network, the relationship corresponding to the text semantic feature calculation process is: ; in, Represents the semantic features of text, Represents the input text sequence, represents the semantic temporal feature extraction operation, Represents the text semantic feature extraction operation.

3. According to claim 2, the multimodal rumor detection method based on dual evidence enhancement and image-text similarity perception is characterized in that: In the step of extracting features from the post image using the pre-trained visual model to obtain the image visual features of the post image, the corresponding relationship for the calculation process of the image visual features is: ； in, Represents the visual features of the image, represents the visual feature extraction operation of the image, Represents the input image.

4. According to claim 3, the multimodal rumor detection method based on dual evidence enhancement and image-text similarity perception is characterized in that: In step 2, in the step of performing difference analysis on the post image using the error level analysis method to obtain a difference analysis image, and extracting tampering features from the difference analysis image using the pre-trained visual model to obtain image tampering features, the calculation process of the image tampering features corresponds to the relationship between: ； in, represents the ELA image tampering feature, represents the input ELA image, Indicates taking the average value of all block features.

5. According to claim 4, the multimodal rumor detection method based on dual evidence enhancement and image-text similarity perception is characterized in that: In step 4, in the step of interactively aligning the text screening evidence and the image screening evidence through a multi-head cross attention mechanism to obtain aligned text evidence features and aligned image evidence features, and verifying the consistency between the aligned text evidence features and the aligned image evidence features and the input post, the calculation process of verifying the consistency between the text screening evidence and the image screening evidence and the input post corresponds to the relationship between: ； in, represents the input sequence, Representation and Different input sequences, represents the multi-head cross attention mechanism operation, Represents a splicing operation, represents the first attention head, Indicates Attention head, represents the parameter matrix that needs to be learned, Indicates Attention head, represents the cross attention mechanism operation, Indicates a query, Indicates the key, Indicates the value, Indicates The weight matrix of the attention head, Indicates The weight matrix of the attention head, Indicates The weight matrix of the attention head, represents the normalization operation, Represents the key-value dimension, represents the transpose key, Represents the transpose of a matrix.

6. According to claim 5, the multimodal rumor detection method based on dual evidence enhancement and image-text similarity perception is characterized in that: In step 7, using the graph attention network to perform graph-text semantic matching on the knowledge-enhanced visual scene graph and the knowledge-enhanced text scene graph to obtain the graph-text semantic matching result of the visual scene graph and the text scene graph specifically includes the following sub-steps: The knowledge-enhanced visual scene graph and the text scene graph are sequentially input into the graph attention network, and the relationship features are introduced for mechanism processing to learn the knowledge-enhanced visual scene graph and the text scene graph to obtain the learned visual scene graph and the learned text scene graph; The feature vectors of all nodes in the learned visual scene graph and the learned text scene graph are processed by the average pooling mechanism in turn to obtain the whole-image features of the visual scene graph and the whole-image features of the text scene graph respectively; The whole-image features of the visual scene graph and the whole-image features of the text scene graph are concatenated and input into a fully connected neural network to obtain the image-text semantic matching results of the visual scene graph and the text scene graph.

7. The multimodal rumor detection method based on dual evidence enhancement and image-text similarity perception according to claim 6 is characterized in that: In the step of sequentially inputting the knowledge-enhanced visual scene graph and the text scene graph into the graph attention network, and introducing the relational features for mechanism processing to learn the knowledge-enhanced visual scene graph and the text scene graph to obtain the learned visual scene graph and the learned text scene graph, the corresponding relationship between the calculation process of the learned visual scene graph and the learned text scene graph is: ； in, represents the graph attention mechanism operation, represents the updated features, Represents a splicing operation, represents the number of multi-head attention, Representation Node The set of neighboring nodes of Indicated in Adjacent nodes in attention type For Node The attention weight, The embedding representation of the node is the learning parameter matrix, The embedding representation of the edge is the learning parameter matrix, represents the residual structure parameter matrix, Representation Node The characteristic representation of Representation Node The characteristic representation of Represents edge The characteristic representation of represents the normalization operation, represents the activation function, represents the transpose of the attention weights.

8. The multimodal rumor detection method based on dual evidence enhancement and image-text similarity perception according to claim 7 is characterized in that: In the step of splicing the whole image features of the visual scene graph with the whole image features of the text scene graph and inputting them into the fully connected neural network to obtain the image-text semantic matching results of the visual scene graph and the text scene graph, the calculation process of the image-text semantic matching results of the visual scene graph and the text scene graph corresponds to the relationship between: ； in, Represents the final whole image features, represents the visual scene graph, represents the initial features of the node, represents the initial features of the edge, Represents the final whole image features, Represents a text scene graph, Represents the semantic matching features of images and texts, represents the activation function, represents the parameter matrix that can be learned.

9. The multimodal rumor detection method based on dual evidence enhancement and image-text similarity perception according to claim 8 is characterized in that: In step 8, the feature hierarchical fusion algorithm is used to adaptively fuse the semantic features of the text, the aligned text evidence features, the image visual features, the aligned picture evidence features, the image tampering features and the image-text semantic matching features to obtain the hierarchical final fusion result, which specifically includes the following sub-steps: The feature hierarchical fusion algorithm is used to adaptively fuse text semantic features with aligned text evidence features to obtain the first-layer feature fusion result; Among them, the calculation process of the first layer feature fusion result corresponds to the following relationship: ； ； in, represents the fusion weight, represents the activation function, Represents a splicing operation, Represents the semantic features of text, Characteristic of textual evidence representing alignment, represents the second parameter matrix to be learned, Represents the first layer feature fusion result; The feature hierarchical fusion algorithm is used to adaptively fuse the image visual features with the aligned picture evidence features to obtain the second-layer feature fusion result; Among them, the corresponding relationship between the calculation process of the second-layer feature fusion result is as follows: ； in, represents the third parameter matrix to be learned, Represents the visual features of the image, Represents the aligned image evidence features, Represents the second layer feature fusion result; The feature layered fusion algorithm is used to adaptively fuse the image tampering features with the image-text semantic matching features to obtain the third-layer feature fusion result; Among them, the calculation process of the third-layer feature fusion result corresponds to the following relationship: ； in, represents the third-layer feature fusion result, represents the fourth parameter matrix to be learned, Represents the semantic matching features of images and texts, Indicates the characteristics of image tampering; The feature layered fusion algorithm is used to adaptively fuse the first layer feature fusion result with the second layer feature fusion result and the third layer feature fusion result to obtain the final fusion result; Among them, the calculation process of the third-layer feature fusion result corresponds to the following relationship: ; in, represents the final fusion result, represents the fifth parameter matrix to be learned, represents the sixth parameter matrix to be learned, represents the seventh parameter matrix to be learned, represents the second fusion weight, represents the third fusion weight, Represents the fourth fusion weight.

10. The multimodal rumor detection method based on dual evidence enhancement and image-text similarity perception according to claim 9 is characterized in that: In step 9, in the step of inputting the final fusion result of the layers into the multilayer perceptron to obtain the result classification, the calculation process of obtaining the result classification corresponds to the relationship: ； in, Represents the predicted output result, represents the parameters of the linear layer, represents the final fusion result, represents the bias top of the linear layer, Represents the operation of the multi-layer perceptron mechanism.

Images