False information intelligent detection and traceability method based on cross-modal consistency verification

Abstract

The invention discloses a false information intelligent detection and traceability method based on cross-modal consistency verification. The method specifically comprises the following steps: S1, multi-modal content feature extraction; s2, carrying out cross-modal consistency verification; s3, performing deep counterfeiting detection; s4, propagation anomaly detection; s5, checking knowledge enhancement facts; s6, information tracing and variation tracking; s7, comprehensive authenticity evaluation; s8, a model training strategy; a multi-task learning framework is adopted, and parameters of all modules are jointly optimized. According to the method, a multi-dimensional consistency verification system is constructed by fusing multi-source information such as video content, text semantics, a propagation network and an external knowledge base, and automatic identification, authenticity evaluation and information traceability of various false information are realized.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, multimedia information processing, and information security technology, specifically to an intelligent detection and tracing method for false information based on cross-modal consistency verification, applicable to various application scenarios such as content review on social media platforms, fact-checking in news media, public opinion monitoring and early warning, and assistance in judicial evidence collection. Background Technology

[0002] With the rapid development of the internet and social media, the spread of misinformation has become a serious social problem. Misinformation includes not only traditional text-based rumors, but also various forms such as deepfake videos, manipulated images, and content taken out of context. This misinformation spreads rapidly and has a wide impact, causing serious harm to individuals, businesses, and society.

[0003] Current misinformation detection technologies mainly face the following problems:

[0004] 1. Limitations of Single-Modal Detection: Most existing methods only target a single modality (such as detecting only text or only images), and cannot effectively handle multimodal spoofing information such as video-text combinations and image-title combinations. For example, inconsistent spoofing information across modalities, such as real videos with fake titles or normal images with misleading text, is difficult to identify by single-modal methods.

[0005] 2. Insufficient deepfake detection capabilities: With the development of technologies such as Generative Adversarial Networks (GANs) and diffusion models, the quality of deepfake videos and images is constantly improving, making it difficult for traditional detection methods to identify high-quality forged content. Existing methods mainly rely on artifact detection at the technical level, lacking semantic-level logical consistency verification.

[0006] 3. Lack of analysis on propagation patterns: False information often spreads through abnormal propagation patterns, such as bulk forwarding by bot accounts and coordinated rumor-mongering by online trolls. Existing methods mainly focus on the content itself, lacking sufficient analysis of the structural characteristics and temporal patterns of the propagation network, and are unable to identify human-manipulated propagation behaviors.

[0007] 4. Inefficient Fact Checking: Traditional fact checking relies on manual verification, which is inefficient, costly, and unable to meet the real-time verification needs of massive amounts of information. Existing automated fact checking methods are mainly based on keyword matching and simple rules, which cannot understand complex semantics and contextual relationships.

[0008] 5. Difficulty in tracing the source of information: False information is often mutated during its dissemination through editing, dubbing, and title changes, making it difficult to trace its original source. Existing source tracing methods are mainly based on simple text similarity or image hashing, which cannot handle complex content mutations and cross-platform dissemination.

[0009] 6. Insufficient fusion of multi-source information: Detection of false information requires the comprehensive use of multiple information sources such as content features, dissemination features, user features, and external knowledge bases. However, existing methods lack an effective mechanism for fusion of multi-source information, and there is a lack of collaborative verification among the various information sources.

[0010] Therefore, there is an urgent need for an intelligent method that can integrate multimodal content, propagation networks, and external knowledge to achieve automatic detection, authenticity assessment, and information tracing of misinformation through cross-modal consistency verification and multi-dimensional analysis. This invention addresses the aforementioned technical challenges by constructing a cross-modal consistency verification model, a propagation anomaly detection model, and a knowledge-enhanced reasoning model, providing an efficient and reliable technical solution for misinformation governance. Summary of the Invention

[0011] This invention provides a method for intelligent detection and tracing of false information based on cross-modal consistency verification. By integrating multi-source information such as video content, text semantics, propagation networks and external knowledge bases, a multi-dimensional consistency verification system is constructed to achieve automatic identification, authenticity assessment and information tracing of various types of false information.

[0012] The specific plan is as follows:

[0013] A method for intelligent detection and tracing of false information based on cross-modal consistency verification includes the following steps:

[0014] S1. Multimodal content feature extraction: Design specialized feature extractors for video, image and text modalities to extract semantic and structural features at the content level;

[0015] S2, Cross-modal consistency verification: Based on the extracted multimodal content features, a cross-modal semantic alignment and consistency measurement mechanism is designed to detect the consistency between video content and text descriptions, images and titles, and other cross-modal combinations; this module is a key step in identifying false information such as "real videos with fake titles";

[0016] S3, Deepfake Detection; Based on the extracted video content features, deepfake detection is performed; It not only detects forgery traces at the technical level, but also analyzes the logical consistency at the semantic level, constructs a multi-dimensional chain of forgery evidence, complements cross-modal consistency verification, and jointly identifies various types of false information at the content level;

[0017] S4. Anomaly Detection: After completing content-level detection (cross-modal consistency verification and deepfake detection), it further analyzes the structural characteristics and temporal patterns of the information dissemination network to identify abnormal dissemination behaviors; it works in conjunction with content detection to provide important evidence for the determination of false information from the dissemination dimension;

[0018] S5. Knowledge-enhanced fact-checking: Based on the extracted semantic features of the text and the original text content, knowledge-enhanced fact-checking is implemented; the extracted entities, events, and relationships are aligned with external knowledge bases to verify factual claims, forming a multi-dimensional verification system with content detection and dissemination analysis;

[0019] S6. Information source tracing and mutation tracking; Based on the extracted video spatial features, information source tracing and mutation tracking are implemented; Through content fingerprinting, similarity matching and propagation tree analysis, the source of information is traced and the mutation process is tracked, providing source tracing evidence for the determination of false information, which together with the aforementioned detection constitutes a complete detection and source tracing system;

[0020] S7. Comprehensive Authenticity Assessment: Integrating the aforementioned outputs, a comprehensive authenticity assessment is performed. This is the system's final decision-making layer. By fusing detection results from cross-modal consistency verification, deepfake detection, propagation anomaly detection, and knowledge-enhanced fact-checking, a comprehensive authenticity score is calculated. First, the physical consistency loss and propagation anomaly indicators are normalized, and then the final score is obtained through weighted fusion.

[0021] S8. Model training strategy: A multi-task learning framework is adopted to jointly optimize the parameters of each module.

[0022] Furthermore, step S1 specifically includes:

[0023] S11. Video Content Feature Extraction

[0024] For the input video, a temporal convolutional neural network combined with a visual Transformer is used to extract multi-scale spatiotemporal features; let the input video be... ,in Indicates the first Frame image, This represents the total number of frames in the video. and These are the image height and width, respectively.

[0025] First, a visual encoder is used to extract spatial features from each frame. The visual encoder is implemented using a deep convolutional neural network (such as ResNet, EfficientNet, etc.), which extracts deep semantic features of the image through multiple convolution and pooling operations.

[0026]

[0027] in For visual features, a temporal convolutional network is used to extract temporal features in order to capture the temporal dynamic features of the video. The temporal convolutional network uses 3D convolutional kernels to perform convolution operations in the time dimension, fusing spatial features of adjacent frames to capture the temporal changes of actions and scenes.

[0028]

[0029] in The window size for temporal convolution. The feature vector is fused with temporal information; finally, global video features are aggregated through an attention mechanism; the attention mechanism adopts the soft attention method, which adaptively aggregates features from all frames by learning the importance weight of each frame:

[0030]

[0031] in The weight matrix is ​​a learnable matrix. For attention weight vectors, For the first Attention score of a frame This is the aggregated video content feature vector;

[0032] S12. Text Semantic Feature Extraction

[0033] For input text (such as video titles, captions, comments, etc.), a pre-trained language model is used to extract deep semantic features; assuming the input text is segmented into word sequences... ,in For word count;

[0034] Contextual representations are obtained through a pre-trained language model; the text encoder is implemented using a pre-trained language model with a Transformer architecture (such as BERT, RoBERTa, etc.), which generates an embedding vector containing contextual information for each word through self-attention mechanism and positional encoding.

[0035]

[0036] in For the first The context embedding vector of each word (obtained from the output of the encoding layer of a pre-trained language model). For text feature dimensions, This is the semantic feature vector of the text after average pooling; the average pooling operation averages the embedding vectors of all words to obtain the global semantic representation of the entire text.

[0037] S13, Image Content Feature Extraction

[0038] For static images, the same visual encoder used for videos is employed to extract spatial features:

[0039]

[0040] in For the input image, This is the image content feature vector.

[0041] Furthermore, step S2 specifically includes:

[0042] S21, Cross-modal semantic alignment

[0043] Features from different modalities are mapped to a unified semantic space for alignment; for video-text pairs, video features and text features are projected onto a common semantic space respectively.

[0044]

[0045] in and For a learnable projection matrix, To unify the semantic space dimension, These are the projected feature vectors;

[0046] S22, Consistency Measurement

[0047] Calculate the similarity of cross-modal features in the semantic space as a consistency metric:

[0048]

[0049] in Describes the L2 norm of a vector. This represents the cross-modal consistency score; a higher value indicates greater consistency between the video content and the text description. Below the preset threshold When cross-modal inconsistencies are found, it may indicate false information (such as a real video with a false title).

[0050] Furthermore, step S3 specifically includes:

[0051] S31, Technical level forgery detection

[0052] Frequency domain analysis and deep neural networks are used to detect traces of deepfake techniques; for video frames Extracting frequency domain features:

[0053]

[0054] in Represents the Fast Fourier Transform. The frequency domain representation is used; frequency domain anomalies are detected by a trained classifier; the classifier is implemented using a deep convolutional neural network (such as Xception, EfficientNet, etc.), which concatenates the frequency domain features and spatial features and inputs them into the classifier. After passing through multiple convolutional and fully connected layers, the forgery probability is output.

[0055]

[0056] in For deep fake classifiers, frequency domain features are used... and spatial features Feature fusion (such as concatenation or weighted summation) is performed, followed by processing through a multi-layer neural network, and finally output through the Sigmoid activation function. To fabricate probabilities at the technical level;

[0057] S32, Semantic Level Logic Verification

[0058] The analysis examines the physical consistency (lighting, shadows, reflections) and temporal consistency (motion coherence, lip-sync) of video content. For each physical consistency check, a corresponding calculation method is employed: lighting consistency is measured by analyzing the consistency of light intensity and direction in different areas of the video frame; shadow consistency is measured by detecting whether the direction, length, and shape of shadows conform to physical laws; reflection consistency is measured by analyzing whether the reflection angle and intensity of reflective surfaces conform to optical principles; motion coherence is measured by analyzing the smoothness and continuity of motion between adjacent frames; and lip-sync is measured by detecting the temporal alignment between the speech signal and lip movements. The physical consistency loss is defined as follows:

[0059]

[0060] in The number of physical consistency check items (such as lighting consistency, shadow consistency, reflection consistency, motion continuity, lip-sync, etc.). For the first The weighting coefficients of the item checks satisfy the following conditions: , For the first The inconsistency measure function for the item (obtained through the corresponding calculation method described above, with a value range of...) (A larger value indicates a higher degree of inconsistency). For input video; when Exceeding the preset threshold At that time, it was determined that there was a physical inconsistency, which may indicate that the content was forged.

[0061] Furthermore, step S4 specifically includes:

[0062] S41. Construction of the propagation network

[0063] Modeling the information dissemination process as a directed graph ,in For a set of nodes (users). For the set of edges (forwarding relationships); each node Having feature vectors (Including user activity, number of followers, historical behavior, etc.);

[0064] S42. Anomaly Propagation Pattern Recognition

[0065] A graph neural network is used to analyze the topology of the propagation network and identify abnormal patterns. The graph neural network is implemented using a graph convolutional network (GCN) or a graph attention network (GAT), aggregating features of neighboring nodes through a message passing mechanism to learn node representation vectors, which are then used for anomaly detection. Specifically, the proportion of robot accounts... By analyzing user characteristics (such as registration time, number of followers, historical behavior, etc.), a classifier is used to identify bot accounts, and the ratio of the number of bot accounts to the total number of accounts is calculated; time concentration. By statistically analyzing the forwarding time distribution, the proportion of forwarded requests within a short period (e.g., within 1 hour) to the total number of forwarded requests is calculated; topology anomaly degree. By analyzing the topological characteristics of the propagation network (such as degree distribution, clustering coefficient, path length, etc.), anomaly patterns such as star topologies and abnormal clustering are detected. Anomaly detection algorithms (such as isolated forests and local anomaly factors) are used to calculate anomaly scores; propagation anomaly indices are defined.

[0066]

[0067] in The proportion of robot accounts (calculated using the classification method described above). This represents the time concentration (calculated using the statistical methods described above). This represents the topological anomaly degree (calculated using the anomaly detection method described above; a larger value indicates a higher degree of anomaly). Let be the weighting coefficient, satisfying ;when Exceeding the threshold At that time, it was determined that there was abnormal transmission, which may have been manipulated by humans.

[0068] Furthermore, step S5 specifically includes:

[0069] S51. Entity and Relationship Extraction

[0070] This process extracts entity and relation sets from text and video. For text, Named Entity Recognition (NER) is used, employing pre-trained sequence labeling models (such as BERT-CRF and BiLSTM-CRF) to identify entities like names of people, places, organizations, times, and events. Relation extraction techniques are used, employing pre-trained relation classification models (such as BERT and RoBERTa) to identify relationships between entities (e.g., "occurred," "located in," "belongs to"). For video, visual entity recognition (VID) is used, employing object detection and image classification models to identify visual entities such as objects, scenes, and people. Video relation extraction techniques are used, analyzing the spatial and temporal relationships between entities in the video frame sequence to identify relationships between visual entities. The final entity set is then obtained. and relation set ,in The number of entities extracted. The number of relations extracted;

[0071] S52. Knowledge Base Alignment and Validation

[0072] The extracted entities and relationships are integrated with the knowledge graph. Alignment is performed, and the credibility of the facts is calculated:

[0073]

[0074] in and These are the sets of entities and relations in a knowledge graph, respectively. Indicates the size of the set. For fact credibility score; when If the value is below the threshold, a factual error is determined to exist.

[0075] Furthermore, step S6 specifically includes:

[0076] S61, Content Fingerprint Generation

[0077] Generate robust content fingerprints for videos and text. For videos, firstly, keyframe extraction algorithms (such as scene change detection or content complexity-based methods) are used to select representative keyframes. Then, spatial features of the keyframes are extracted. Finally, a visual fingerprint is generated using perceptual hashing or deep hashing. Perceptual hashing generates fingerprints robust to slight changes by mapping image features to fixed-length binary codes. Deep hashing learns a hash function by training a deep neural network, mapping high-dimensional features to low-dimensional hash codes. For video fingerprints, features from multiple keyframes are fused (e.g., using average pooling or max pooling) and then generated using a hash function.

[0078]

[0079] in For keyframe indexing, For hash functions (implemented using perceptual hashing or deep hashing methods). It is a video fingerprint vector (usually a 128-dimensional or 256-dimensional binary code or floating-point vector).

[0080] S62. Similarity Matching and Source Tracing

[0081] Searching for similar content in historical databases, similar fingerprints are quickly retrieved using approximate nearest neighbor search algorithms (such as LSH, Annoy, Faiss, etc.), and then the precise similarity is calculated. The similarity calculation uses the cosine similarity method to measure the angle between two fingerprint vectors in the vector space.

[0082]

[0083] in For the fingerprint vector of the query content, This is a fingerprint vector of historical content in the database. Describes the L2 norm of a vector. The similarity score is used to identify potential sources when the similarity exceeds a preset threshold (e.g., 0.6). The earliest publication time is determined through propagation tree analysis to locate the information source. Propagation tree analysis constructs a directed graph of information propagation, backtracks from the current node to all possible source nodes, compares the publication times of each source node, and selects the node with the earliest publication time as the information source. The propagation tree is traversed using depth-first search (DFS) or breadth-first search (BFS) algorithms, and the earliest publication node is determined by combining the timestamp information.

[0084] Furthermore, step S7 specifically includes:

[0085]

[0086] in The fusion weight coefficients for each output satisfy the following conditions: , and These are the maximum normalization factors for physical consistency loss and propagation anomaly index, respectively. The cross-modal consistency score (calculated by the cross-modal consistency verification module). The probability of forgery at the technical level (calculated by the technical level detection submodule of the deepfake detection module). The physical consistency loss is calculated by the semantic level verification submodule of the deepfake detection module. This is a propagation anomaly indicator (calculated by the propagation anomaly detection module). The fact credibility score (calculated by the knowledge-enhanced fact-checking module); To assess the overall authenticity score, a higher value indicates more credible information; when... Below the preset threshold At that time, it was determined to be false information;

[0087] Furthermore, step S8 specifically includes:

[0088] The total loss function is defined as:

[0089]

[0090] in For consistency verification loss, In order to fabricate testing losses, To prevent the loss of anomaly detection, To verify the loss, To trace the source of the loss, These are the weighting coefficients for each loss.

[0091] Through the above technical solutions, this invention constructs a complete system for detecting and tracing false information. It can comprehensively utilize content features, dissemination features, and external knowledge, and through cross-modal consistency verification and multi-dimensional analysis, it can accurately identify and effectively trace the source of various types of false information.

[0092] The beneficial effects of this invention are as follows: This method integrates multi-source information such as video content, text semantics, propagation networks, and external knowledge bases to construct a multi-dimensional consistency verification model, enabling automatic identification, authenticity assessment, and information tracing of various types of false information, including deepfakes, content tampering, and false narratives. Specific advantages are as follows:

[0093] 1. By integrating multimodal content, dissemination networks, and external knowledge, this invention improves the accuracy of false information detection by 20-25 percentage points compared to single-modal detection methods, and can more accurately identify various types of false information, including deepfakes, content tampering, false narratives, and other forms.

[0094] 2. This invention innovatively designs a cross-modal consistency verification mechanism, capable of identifying cross-modal inconsistent false information such as "real videos with fake titles" and "normal images with misleading text," a capability that existing single-modal detection methods cannot achieve. This capability has significant application value in social media content moderation scenarios.

[0095] 3. This invention not only detects forgery traces at the technical level but also analyzes semantic logical consistency, constructing a multi-dimensional chain of forgery evidence. Compared to traditional methods that rely solely on technical detection, this invention can handle higher-quality deepfake content, improving detection accuracy by 15-20 percentage points.

[0096] 4. By analyzing the characteristics of the propagation network through graph neural networks, it can identify abnormal propagation behaviors such as paid online commenters, fake followers forwarding content, and coordinated rumor-mongering, distinguishing between natural propagation and human manipulation. This capability provides an important basis for judging false information, with an F1 score of over 85% for detecting propagation anomalies.

[0097] 5. The extracted entities and relationships are aligned with external knowledge bases to verify the factual accuracy of the information, detecting factual errors, spatiotemporal contradictions, and logical conflicts. Compared to simple keyword matching methods, the fact-checking accuracy of this invention is improved by 30-35 percentage points.

[0098] 6. A robust content fingerprinting method was designed to resist content mutation. Through similarity matching and propagation tree analysis, the source of information can be traced and the mutation process can be tracked. The source tracing success rate reaches over 79%, providing important technical support for the governance of misinformation.

[0099] 7. This invention constructs a complete multi-module collaborative system with clear connections between modules, allowing for flexible combination and use. The system has good scalability, easily integrating new detection modules or knowledge sources to adapt to the needs of different application scenarios.

[0100] 8. This invention can be applied to multiple fields such as content review on social media platforms, fact-checking in news media, public opinion monitoring and early warning, judicial evidence collection assistance, financial market supervision, and public health emergency response, and has broad market application prospects and important social value.

[0101] 9. This invention integrates cutting-edge technologies such as multimodal learning, graph neural networks, knowledge graphs, and information tracing, and proposes innovative methods such as cross-modal consistency verification, multi-dimensional forgery detection, and propagation anomaly identification, which have significant advancements and innovations at the technical level.

[0102] 10. This invention is based on a mature deep learning framework and pre-trained model. It does not require dedicated hardware and can be deployed on a standard server, which reduces the system deployment cost and the threshold for use, and is conducive to the rapid promotion and practical application of the technology. Attached Figure Description

[0103] Figure 1 This is a flowchart of the method of the present invention.

[0104] Figure 2 This is a framework diagram of the present invention. Detailed Implementation

[0105] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0106] As shown in the figure, this embodiment provides a method for intelligent detection and tracing of false information based on cross-modal consistency verification, taking video content review on a social media platform as an application scenario. The specific solution is as follows:

[0107] I. System Architecture and Data Acquisition

[0108] A fake information detection system is constructed, which includes a data acquisition module, a multimodal feature extraction module, a consistency verification module, a forgery detection module, a propagation analysis module, a knowledge verification module, a source tracing module, and a comprehensive evaluation module.

[0109] Data collection configuration: Video data: Collect videos to be tested from social media platforms, with a resolution of no less than 720p and a duration of 3-60 seconds; Text data: Collect text information such as video titles, captions, and comments; Dissemination data: Collect dissemination network data such as forwarding relationships, forwarding times, and user information; Knowledge base: Integrate external knowledge sources such as encyclopedia knowledge bases, news databases, and historical debunking databases.

[0110] The system performs a comprehensive analysis on each unit of information to be detected (video + text + dissemination data) and outputs the authenticity assessment results and the source tracing report.

[0111] II. Implementation of Multimodal Content Feature Extraction

[0112] Based on the aforementioned multimodal content feature extraction method, this embodiment specifically implements the feature extraction process for video and text. This step is the foundation of the entire detection process, and the extracted features will be used in subsequent modules such as consistency verification and forgery detection.

[0113] (1) Video content feature extraction

[0114] The input video undergoes preprocessing, including frame extraction, face detection, and key region localization. Assuming the video is 10 seconds long and has a frame rate of 30fps, a total of 300 frames are extracted. Based on the aforementioned video content feature extraction method, ResNet-50 is used as the visual encoder (i.e., the specific implementation of VisualEncoder) to extract the spatial features of each frame:

[0115]

[0116] in For the first Frame image, visual feature dimension Based on the aforementioned temporal feature extraction method, a 3D convolutional network (i.e., a specific implementation of TemporalConv) is used to extract temporal features, with a window size of... (Corresponding to 5 frames:) ):

[0117]

[0118] Based on the aforementioned attention aggregation method, global features are aggregated through a temporal attention mechanism. An attention weight matrix is ​​then set. and attention weight vector Calculate the attention score for each frame and aggregate them:

[0119]

[0120] Obtain video content features This feature will be used for subsequent cross-modal consistency verification and deepfake detection.

[0121] (2) Extraction of semantic features of text

[0122] The video title and text undergo preprocessing, including word segmentation and stop word removal. For example, assuming the title is "A major accident occurred in a certain place, the scene was horrific," word segmentation yields a word sequence. , number of words Based on the aforementioned text feature extraction methods, the BERT-base model is used as the text encoder (i.e., the specific implementation of TextEncoder):

[0123]

[0124] in For the first The context embedding vector of each word (BERT-base hidden layer dimension, i.e., text feature dimension) (to obtain text semantic features) This feature will be used together with video features for cross-modal consistency verification.

[0125] III. Implementation of Cross-Modal Consistency Verification

[0126] Based on the extracted video and text features, this embodiment implements cross-modal consistency verification, a crucial step in detecting false information such as "real video with fake title." This module is based on the aforementioned cross-modal consistency verification method.

[0127] (1) Feature projection

[0128] Based on the aforementioned cross-modal semantic alignment method, the obtained video features are... and text features Projected onto a unified 512-dimensional semantic space ( ):

[0129]

[0130] in For video feature projection matrix, The text feature projection matrix, This is the projected feature vector. The projected features will be used to calculate the consistency score.

[0131] (2) Consistency calculation

[0132] Based on the aforementioned consistency measurement method, the similarity of cross-modal features in the semantic space is calculated. Suppose that in a certain detection, the video content shows a normal street scene, but the title claims "a major accident has occurred," the calculation is as follows:

[0133]

[0134] in Describes the L2 norm of a vector. Cross-modal consistency score. Set a threshold. ,because The system determined that there was cross-modal inconsistency, and the information might be false (a real video with a false title). This consistency score... It will be passed as input to the comprehensive authenticity assessment module.

[0135] IV. Implementation of Deepfake Detection

[0136] Based on the video features extracted above, this embodiment implements deepfake detection, which is a crucial step in identifying technically forged content. This module, based on the aforementioned deepfake detection method, includes detection at both the technical and semantic levels.

[0137] (1) Technical testing

[0138] Based on the aforementioned forgery detection methods, frequency domain analysis is performed on the video frames. Let's assume we're analyzing the 100th frame... Perform a Fast Fourier Transform:

[0139]

[0140] in Represents the Fast Fourier Transform. This is represented in the frequency domain. Combined with the spatial features extracted above... Through a well-trained deep fake classifier (based on the Xception network, i.e., Classifier) (Specific implementation) Detection:

[0141]

[0142] in To fabricate probabilities at a technical level. Because... (Judgment threshold) No obvious signs of forgery were detected from a technical standpoint. This forgery probability... It will be passed as input to the comprehensive authenticity assessment module.

[0143] (2) Semantic level verification

[0144] Based on the aforementioned semantic-level logical verification method, the physical consistency of the video is analyzed. The methods described above are used: illumination consistency is measured by analyzing the consistency of illumination intensity and direction in different regions of the video frame, resulting in an inconsistency value of 0.12; shadow consistency is measured by detecting whether the direction, length, and shape of shadows conform to physical laws, resulting in an inconsistency value of 0.08; reflection consistency is measured by analyzing whether the reflection angle and intensity of reflective surfaces conform to optical principles, resulting in an inconsistency value of 0.15. Assuming these three items are checked (… The weighting coefficients are set to... Based on the aforementioned formula for physical consistency loss, the following calculation is obtained:

[0145]

[0146] in To input video, For the first Inconsistency measure for items. Set a threshold. ,because The physical consistency check passed. The physical consistency loss... It will be passed as input to the comprehensive authenticity assessment module.

[0147] V. Implementation of Anomaly Detection

[0148] This embodiment implements propagation anomaly detection, a crucial step in identifying human manipulation of propagation behavior. Based on the aforementioned propagation anomaly detection method, this module identifies abnormal behaviors such as paid online commenters and fake followers forwarding content by analyzing the structural characteristics and temporal patterns of the propagation network.

[0149] (1) Construction of the propagation network

[0150] Based on the aforementioned propagation network construction method, assuming a message is forwarded 1000 times, construct the propagation network. , where the set of nodes Contains 1000 nodes (forwarding users), edge set It contains 1000 edges (forwarding relationships). Extract each node. eigenvectors (Including registration time, number of followers, historical forwarding behavior, etc.) for subsequent abnormal pattern identification.

[0151] (2) Abnormal pattern recognition

[0152] Based on the aforementioned abnormal propagation pattern identification method, a Graph Neural Network (GCN) is used to analyze the topology of the propagation network. Various indicators are calculated using the methods described above: the proportion of bot accounts is determined by analyzing user characteristics (registration time, number of followers, historical behavior, etc.) and using a classifier to identify bot accounts, calculating the ratio of the number of bot accounts to the total number of accounts. (65% are bot accounts); Time concentration is obtained by statistically analyzing the distribution of forwarding time and calculating the proportion of forwards within one hour to the total number of forwards. (82% of forwardings occurred within 1 hour); Topology anomaly detection was achieved by analyzing the topological characteristics of the propagation network (degree distribution, clustering coefficients, etc.) and using the Isolation Forest algorithm to detect abnormal patterns. (It exhibits a distinct star-shaped structure, indicating a high degree of anomaly).

[0153] Based on the aforementioned formula for the propagation anomaly index, weighting coefficients are set. The calculation yields:

[0154]

[0155] in Let be the weighting coefficient, satisfying Set a threshold. ,because The system detected an abnormal propagation pattern, potentially indicating the spread of misinformation manipulated by individuals. This abnormal propagation indicator... It will be passed as input to the comprehensive authenticity assessment module.

[0156] VI. Implementation of Knowledge-Enhanced Fact-Checking

[0157] Based on the extracted text features and original text content, this embodiment implements knowledge-enhanced fact-checking, a crucial step in verifying the factual nature of information. This module, based on the aforementioned knowledge-enhanced fact-checking method, verifies factual claims by aligning the extracted entities and relationships with an external knowledge base.

[0158] (1) Entity and relation extraction

[0159] Based on the aforementioned entity and relation extraction methods, entity and relation sets are extracted from the text "A major accident occurred in a certain place, and the scene was horrific." The methods described above are employed: BERT-CRF model is used for named entity recognition to identify the place name entity "a certain place" and the event entity "major accident"; BERT relation classification model is used for relation extraction to identify the "occurrence" relationship between "a certain place" and "major accident." Assuming the extracted entity set... (Number of entities) ), set of relations (Number of relations) These entities and relationships will be used for alignment validation with the knowledge base.

[0160] (2) Knowledge base verification

[0161] Based on the aforementioned knowledge base alignment and verification methods, the extracted entities and relationships are aligned with the knowledge graph. Alignment is performed. Suppose we are querying the knowledge base for information related to "location" and "major accident," and the knowledge base contains the entity "location" (i.e., ...). The intersection size is 1), but no records related to "major accident" were found, and the "occurrence" relationship does not exist in the knowledge base (i.e., (The intersection size is 0). Based on the aforementioned formula for the credibility of facts, the following is calculated:

[0162]

[0163] in and These are the sets of entities and relations in a knowledge graph, respectively. Indicates the size of the set. This is the credibility score of the facts. Because... The system determined that the information lacked factual support in the knowledge base and posed a risk of factual error. The factual credibility score was... It will be passed as input to the comprehensive authenticity assessment module.

[0164] VII. Implementation of Information Tracing and Mutation Tracking

[0165] Based on the aforementioned extracted video spatial features, this embodiment implements information tracing and mutation tracking, which is an important step in tracing the source of information. This module, based on the aforementioned information tracing and mutation tracking methods, traces the source of information and tracks the mutation process through content fingerprint generation and similarity matching.

[0166] (1) Content fingerprint generation

[0167] Based on the aforementioned content fingerprint generation method, a robust content fingerprint is generated for the video. The method described above is used as follows: First, a keyframe extraction algorithm based on scene change detection is employed, selecting one frame every 2 seconds as a keyframe, for a total of 5 frames extracted. The keyframe index is... ( Then, the extracted keyframe spatial features are used; finally, a deep hashing method is employed to fuse the features of multiple keyframes through average pooling, and then a 128-dimensional fingerprint vector is generated using a trained deep neural network hash function. According to the aforementioned fingerprint generation formula:

[0168]

[0169] in Using a hash function, a 128-dimensional fingerprint vector is obtained. This fingerprint will be used to search for similar content in the historical database.

[0170] (2) Similarity matching

[0171] Based on the aforementioned similarity matching and tracing methods, similar content is searched in a historical database (containing 100,000 historical video records). The method described above is employed: first, the Faisal approximate nearest neighbor search algorithm is used to quickly retrieve similar fingerprints (retrieving the top-100 candidates), and then the precise similarity is calculated. For the query fingerprint... fingerprints in the database According to the aforementioned cosine similarity calculation formula:

[0172]

[0173] Suppose three similar videos are found, with similarity scores of 0.89, 0.76, and 0.68 respectively (all exceeding the similarity threshold of 0.6). Analyzing the publication dates of these three videos, the earliest was published on March 15, 2023, while the currently detected video was published on May 20, 2023. Using the propagation tree analysis method described above: a directed graph of information propagation is constructed, tracing back from the current node (the video published on May 20, 2023) to all possible source nodes. A depth-first search (DFS) algorithm is used to traverse the propagation tree, and the publication dates of each source node are compared using timestamp information to determine that the original video was published on March 15, 2023, and the current video is its variant version (possibly edited or re-dubbed). The source tracing results will be included as evidence in the final comprehensive authenticity assessment report.

[0174] VIII. Implementation of Comprehensive Authenticity Assessment

[0175] By integrating the outputs of the aforementioned detection modules, this embodiment performs a comprehensive authenticity assessment, which is a crucial step in ultimately determining the authenticity of the information. Based on the aforementioned comprehensive authenticity assessment method, this module obtains the final authenticity score by weightedly fusing the detection results of each module.

[0176] Based on the aforementioned comprehensive authenticity assessment formula, a fusion weighting coefficient is set. (satisfy ), normalization factor .

[0177] Based on the output of each module: (From the cross-modal consistency verification module, cross-modal inconsistency). (From the technical detection submodule of the deepfake detection module, no forgery was detected at the technical level.) (From the semantic-level verification submodule of the deepfake detection module, with good physical consistency). (From the propagation anomaly detection module, propagation anomaly); (From the knowledge-enhanced fact-checking module, fact-checking failed)

[0178] Based on the aforementioned formula for comprehensive authenticity score, the following is calculated:

[0179]

[0180] Substitute specific values:

[0181]

[0182] in To determine the overall authenticity score, a threshold is set. ,because The system ultimately determines that the information is false and generates a detailed report, including: cross-modal inconsistency evidence (from the cross-modal consistency verification module), propagation anomaly analysis (from the propagation anomaly detection module), fact-checking results (from the knowledge-enhanced fact-checking module), and source tracing information (from the information source tracing and mutation tracking module). This comprehensive evaluation result integrates the outputs of all detection modules, providing a comprehensive and reliable basis for determining false information.

[0183] IX. Model Training Implementation

[0184] Based on the aforementioned model training strategy, this embodiment details the training process of the multi-task learning framework. This training process optimizes the parameters of all the aforementioned modules, enabling the entire system to work collaboratively and achieve accurate detection and tracing of misinformation.

[0185] Based on the aforementioned multi-task learning framework, a total loss function is used to jointly optimize the parameters of each module. The total loss function is defined as:

[0186]

[0187] in This is the consistency verification loss (corresponding to the cross-modal consistency verification module). This is to prevent the detection loss from being faked (corresponding to the deepfake detection module). This is to reduce the loss from propagation anomaly detection (corresponding to the propagation anomaly detection module). Loss due to fact-checking (corresponding to the knowledge-enhanced fact-checking module). This is for source tracing loss (corresponding to the information source tracing and mutation tracking module). These are the weighting coefficients for each loss.

[0188] Training datasets: Fake information dataset: containing 50,000 labeled fake information samples (videos + text + dissemination data); Real information dataset: containing 100,000 real information samples; Knowledge base: Wikipedia, news databases, historical debunking databases.

[0189] Training configuration: Optimizer: AdamW, learning rate Batch size: 32; Number of training epochs: 20; Loss weights: (satisfy )

[0190] After training, the model's performance metrics on the test set were as follows: False information detection accuracy: 91.5%; Cross-modal consistency verification accuracy: 88.3%; Propagation anomaly detection F1 score: 85.7%; Fact checking accuracy: 82.1%; Source tracing success rate: 79.4%.

[0191] Through the detailed description of this embodiment, those skilled in the art can clearly understand the technical solution of the present invention and can implement the present invention in practical application scenarios according to the described method and system configuration, so as to achieve efficient and accurate detection and tracing of false information. The entire system starts from data collection, goes through multimodal feature extraction, consistency verification, forgery detection, propagation analysis, fact verification, and information tracing, and finally obtains a judgment result through comprehensive evaluation, forming a complete, coherent and interconnected technical process.

[0192] This invention designs a specialized cross-modal semantic alignment and consistency measurement method. By projecting features from different modalities onto a unified semantic space and calculating semantic similarity, it effectively detects inconsistencies between cross-modal combinations such as video content and text descriptions, and images and titles. This method can identify common forms of misinformation such as "real videos with fake titles" and "normal images with misleading text."

[0193] This invention not only detects forgery traces at the technical level (frequency domain anomalies, GAN artifacts, etc.), but also analyzes logical consistency at the semantic level (physical consistency, temporal consistency, etc.), constructing a multi-dimensional chain of forgery evidence. This method can handle high-quality deepfake content, improving detection accuracy and robustness.

[0194] This invention analyzes the structural characteristics and temporal patterns of propagation networks using graph neural networks, quantifying indicators such as the proportion of bot accounts, temporal concentration, and topological anomalies to identify abnormal propagation behaviors such as paid followers inflating engagement metrics, fake followers forwarding content, and coordinated rumor-mongering. This method can distinguish between natural propagation and human manipulation, providing crucial evidence for identifying false information.

[0195] This invention aligns extracted entities, events, and relationships with an external knowledge base and verifies the factual validity of information by calculating factual credibility. This method can detect factual errors, spatiotemporal contradictions, and logical conflicts, improving the accuracy and interpretability of misinformation detection.

[0196] This invention designs a robust content fingerprint generation method that can resist content mutations (editing, dubbing, title changes, etc.). Through similarity matching and propagation tree analysis, it traces the information source and tracks the mutation process. This method can handle complex content mutations and cross-platform propagation, achieving effective information tracing.

[0197] This invention designs a multi-module collaborative comprehensive authenticity assessment framework that integrates content features, dissemination features, and external knowledge, and calculates a comprehensive authenticity score through weighted fusion. This method can fully utilize the complementarity of multi-source information to improve the comprehensiveness and accuracy of misinformation detection.

[0198] This invention employs a multi-task learning framework to jointly optimize tasks such as consistency verification, forgery detection, propagation analysis, fact-checking, and source tracing. By sharing feature representations and joint training, it improves the collaborative effect and overall performance of each module.

[0199] The above embodiments are merely typical illustrative methods of the present invention, and the scope of protection of the present invention is not limited thereto. All equivalent substitutions and improvements made under the concept of the present invention should fall within the scope of protection. It should be emphasized that any modifications or minor adjustments made by those skilled in the art without departing from the basic principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A false information intelligent detection and tracing method based on cross-modal consistency verification, characterized in that, Specifically, the following steps are included: S1. Design specialized feature extractors for video, image and text modalities respectively to extract semantic and structural features at the content level, and realize multimodal content feature extraction; S2. Based on the extracted multimodal content features, design a cross-modal semantic alignment and consistency measurement mechanism to detect the consistency between video content and text description, and between images and titles across modalities. S3. Based on the extracted video content features, implement deepfake detection; not only detect forgery traces at the technical level, but also analyze the logical consistency at the semantic level, construct a multi-dimensional forgery evidence chain, complementing cross-modal consistency verification, and jointly identify various types of false information at the content level; S4. After completing the content-level detection, further analyze the structural characteristics and temporal patterns of the information dissemination network to identify abnormal dissemination behaviors; S5. Based on the extracted text semantic features and original text content, implement knowledge-enhanced fact-checking; align the extracted entities, events, and relationships with external knowledge bases to verify factual claims, forming a multi-dimensional verification system with content detection and dissemination analysis; S6. Based on the extracted video features, implement information tracing and mutation tracking; through content fingerprinting, similarity matching and propagation tree analysis, trace the source of information and track the mutation process, providing tracing evidence for the determination of false information; S7. Integrate the aforementioned outputs and conduct a comprehensive authenticity assessment. First, normalize the physical consistency loss and propagation anomaly indicators, and then obtain the final score through weighted fusion.

2. The method for intelligent detection and tracing of false information based on cross-modal consistency verification according to claim 1, characterized in that, Step S1 specifically includes: S11. Video Content Feature Extraction For the input video, a temporal convolutional neural network combined with a visual Transformer is used to extract multi-scale spatiotemporal features; let the input video be... ,in Indicates the first Frame image, This represents the total number of frames in the video. and These are the image height and width, respectively. First, spatial features of each frame are extracted using a visual encoder. The visual encoder is implemented using a deep convolutional neural network, which extracts deep semantic features of the image through multiple convolution and pooling operations. in For visual features, a temporal convolutional network is used to extract temporal features in order to capture the temporal dynamic features of the video. The temporal convolutional network uses 3D convolutional kernels to perform convolution operations in the time dimension, fusing spatial features of adjacent frames to capture the temporal changes of actions and scenes. in The window size for temporal convolution. The feature vectors are fused with temporal information; finally, global video features are aggregated through an attention mechanism; the attention mechanism adopts a soft attention method, which adaptively aggregates features from all frames by learning the importance weight of each frame. in The weight matrix is ​​a learnable matrix. For attention weight vectors, For the first Attention score of a frame This is the aggregated video content feature vector; S12. Text Semantic Feature Extraction For the input text, a pre-trained language model is used to extract deep semantic features; assuming the input text is segmented into word sequences... ,in For word count; Contextual representations are obtained through a pre-trained language model; the text encoder is implemented using a pre-trained language model based on the Transformer architecture, and generates an embedding vector containing contextual information for each word through self-attention mechanism and positional encoding. in For the first The context embedding vector of each word. For text feature dimensions, This is the semantic feature vector of the text after average pooling; the average pooling operation averages the embedding vectors of all words to obtain the global semantic representation of the entire text. S13, Image Content Feature Extraction For static images, the same visual encoder used for videos is employed to extract spatial features: in For the input image, This is the image content feature vector.

3. The method for intelligent detection and tracing of false information based on cross-modal consistency verification according to claim 1, characterized in that, Step S2 specifically includes: S21, Cross-modal semantic alignment Features from different modalities are mapped to a unified semantic space for alignment; for video-text pairs, video features and text features are projected onto a common semantic space respectively. in and The projection matrix is ​​learnable. To unify the semantic space dimension, These are the projected feature vectors; S22, Consistency Measurement Calculate the similarity of cross-modal features in the semantic space as a consistency metric: in Describes the L2 norm of a vector. This represents the cross-modal consistency score; a higher value indicates greater consistency between the video content and the text description. Below the preset threshold If cross-modal inconsistency is found, it may be false information.

4. The method for intelligent detection and tracing of false information based on cross-modal consistency verification according to claim 1, characterized in that, Step S3 specifically includes: S31, Technical level forgery detection Frequency domain analysis and deep neural networks are used to detect traces of deepfake techniques; for video frames Extracting frequency domain features: in Represents the Fast Fourier Transform. The frequency domain representation is used; frequency domain anomalies are detected by a trained classifier; the classifier is implemented using a deep convolutional neural network, which concatenates frequency domain features and spatial features and inputs them into the classifier. After passing through multiple convolutional and fully connected layers, the forgery probability is output. in For deep fake classifiers, frequency domain features are used... and spatial features Feature fusion is performed, followed by processing through a multi-layer neural network, and finally output through the Sigmoid activation function. To fabricate probabilities at the technical level; S32, Semantic Level Logic Verification The analysis examines the physical and temporal consistency of video content. For each physical consistency check, a corresponding calculation method is employed: illumination consistency is measured by analyzing the consistency of illumination intensity and direction in different regions of the video frame; shadow consistency is measured by detecting whether the direction, length, and shape of shadows conform to physical laws; reflection consistency is measured by analyzing whether the reflection angle and intensity of reflective surfaces conform to optical principles; motion coherence is measured by analyzing the smoothness and continuity of motion between adjacent frames; lip-sync is measured by detecting the temporal alignment between the speech signal and lip movements; the physical consistency loss is defined as follows: in The number of physical consistency check items. For the first The weighting coefficients of the item checks satisfy the following conditions: , For the first Inconsistency measure function of items For input video; when Exceeding the preset threshold At that time, it was determined that there was a physical inconsistency, which may indicate that the content was forged.

5. The method for intelligent detection and tracing of false information based on cross-modal consistency verification according to claim 1, characterized in that, Step S4 specifically includes: S41. Construction of the propagation network Modeling the information dissemination process as a directed graph ,in For a set of nodes, For each node, there is a set of edges. Having feature vectors ; S42. Anomaly Propagation Pattern Recognition A graph neural network is used to analyze the topology of the propagation network and identify abnormal patterns. The graph neural network is implemented using a graph convolutional network (GCN) or a graph attention network (GAT), aggregating features of neighboring nodes through a message passing mechanism to learn node representation vectors, which are then used for anomaly detection. Specifically, the proportion of robot accounts... By analyzing user characteristics and using a classifier to identify bot accounts, the ratio of the number of bot accounts to the total number of accounts is calculated; time concentration. By statistically analyzing the forwarding time distribution, the proportion of forwards within a short period to the total number of forwards is calculated; topology anomaly degree. By analyzing the topological characteristics of the propagation network, anomaly patterns such as star topologies and abnormal clustering are detected, and anomaly scores are calculated using an anomaly detection algorithm; propagation anomaly indicators are defined as follows: in For the proportion of robot accounts, For time concentration, For topological anomaly degree, Let be the weighting coefficient, satisfying ;when Exceeding the threshold At that time, it was determined that there was abnormal transmission, which may have been manipulated by humans.

6. The method for intelligent detection and tracing of false information based on cross-modal consistency verification according to claim 1, characterized in that, Step S5 specifically includes: S51. Entity and Relationship Extraction This process extracts entity sets and relation sets from text and video. For text, Named Entity Recognition (NER) is used, employing a pre-trained sequence labeling model to identify names of people, places, organizations, times, and events. Relation extraction is then used, employing a pre-trained relation classification model to identify relationships between entities. For video, visual entity recognition (VID) is employed, using object detection and image classification models to identify visual entities such as objects, scenes, and people. Video relation extraction is used to analyze the spatial and temporal relationships between entities in a video frame sequence to identify relationships between visual entities. The final entity set is then obtained. and relation set ,in The number of entities extracted. The number of relations extracted; S52. Knowledge Base Alignment and Validation The extracted entities and relationships are integrated with the knowledge graph. Alignment is performed, and the credibility of the facts is calculated: in and These are the sets of entities and relations in a knowledge graph, respectively. Indicates the size of the set. For fact credibility score; when If the value is below the threshold, a factual error is determined to exist.

7. The method for intelligent detection and tracing of false information based on cross-modal consistency verification according to claim 1, characterized in that, Step S6 specifically includes: S61, Content Fingerprint Generation Generate robust content fingerprints for videos and text. For videos, a keyframe extraction algorithm is first used to select representative keyframes, then spatial features of the keyframes are extracted, and finally, perceptual hashing or deep hashing methods are used to generate visual fingerprints. Perceptual hashing generates fingerprints robust to slight changes by mapping image features to fixed-length binary codes. Deep hashing learns a hash function by training a deep neural network to map high-dimensional features to low-dimensional hash codes. For video fingerprints, features from multiple keyframes are fused and then generated using a hash function. in For keyframe indexing, For hash functions, It is a video fingerprint vector; S62. Similarity Matching and Source Tracing The system searches for similar content in the historical database, uses an approximate nearest neighbor search algorithm to quickly retrieve similar fingerprints, and then calculates the precise similarity. The cosine similarity method is used to measure the angle between two fingerprint vectors in the vector space. in For the fingerprint vector of the query content, This is a fingerprint vector of historical content in the database. Describes the L2 norm of a vector. The similarity score is used to identify potential sources when the similarity exceeds a preset threshold. The earliest publication time is determined through propagation tree analysis to locate the information source. Propagation tree analysis constructs a directed graph of information propagation, backtracks from the current node to all possible source nodes, compares the publication times of each source node, and selects the node with the earliest publication time as the information source. The propagation tree is traversed using depth-first search (DFS) or breadth-first search (BFS) algorithms, and the earliest publication node is determined by combining the timestamp information.

8. The method for intelligent detection and tracing of false information based on cross-modal consistency verification according to claim 1, characterized in that, Step S7 specifically includes: in The fusion weight coefficients for each output satisfy the following conditions: , and These are the maximum normalization factors for physical consistency loss and propagation anomaly index, respectively. For cross-modal consistency score, To fabricate probabilities at the technical level, For physical consistency loss, To spread abnormal indicators, This represents the credibility score of the facts. To assess the overall authenticity score, a higher value indicates more credible information; when... Below the preset threshold At that time, it was determined to be false information.

9. The method for intelligent detection and tracing of false information based on cross-modal consistency verification according to claim 1, characterized in that, It also includes the model training strategy in step S8; and adopts a multi-task learning framework to jointly optimize each parameter.

10. The method for intelligent detection and tracing of false information based on cross-modal consistency verification according to claim 9, characterized in that, Step S8 specifically includes: The total loss function is defined as: in For consistency verification loss, In order to fabricate testing losses, To prevent the loss of anomaly detection, To verify the loss, To trace the source of the loss, These are the weighting coefficients for each loss.

Images