Digital multimedia evidence obtaining method and device

Abstract

The invention relates to a digital multimedia evidence obtaining method and device. The digital multimedia evidence obtaining method comprises the steps that a multi-modal knowledge graph is constructed based on multi-modal media samples, and the multi-modal media samples comprise image frames, audio signals and text subtitles; identifying potential forgery risk points from the multi-modal media sample, and generating a forgery task graph for the potential forgery risk points; performing multi-modal feature extraction on the multi-modal media sample based on the forgery task graph to obtain a multi-granularity forgery clue set; structuring the multi-granularity forged clue set into knowledge entries; chain logic reasoning is carried out based on the multi-modal knowledge graph and the knowledge entries, and cross-modal consistency scores are calculated to generate a reasoning tree; determining a final forgery credibility score based on the confidence of each node in the inference tree; and based on the final forgery credibility score, determining whether forgery content exists in the multi-modal media sample. With the adoption of the method and the device, high-precision and high-robustness forged content identification can be realized.

Description

Technical Field

[0001] This disclosure relates to the fields of digital multimedia forensics and deepfake defense technology, and more specifically, to a digital multimedia forensics method and apparatus. Background Technology

[0002] Digital Multimedia Forensics (DMF) is an important multimodal security and AI forensics task. It primarily refers to the process of accurately distinguishing genuine videos / images from forged content when the specific generation process of the forgery is uncertain (e.g., generation model architecture, manipulation methods, compression noise, post-processing operations, etc.). This technology has been widely applied in areas such as social media content moderation, financial identity verification, judicial evidence collection, and cybersecurity. Simultaneously, DMF can also serve as a prerequisite for many advanced security tasks, such as misinformation tracking, identity authentication, and content tracing. The main challenge of DMF tasks is the unknown and complex nature of forgery methods. Real-world digital forgeries often involve the combined effects of different generation models (e.g., GANs, Diffusion), multiple editing operations, and compressed transmission, exhibiting extremely high diversity and concealment. Therefore, effectively characterizing forgery traces and utilizing multimodal consistency features for robust detection is a key focus in this field. Early digital forgery detection methods mostly relied on explicit forgery feature modeling strategies, such as those based on facial keypoint anomalies, frequency domain artifact analysis, or lip-sync inconsistency detection. These methods require first constructing a specific explicit forgery detector, extracting interpretable forgery features, and then inputting them into a classification network. However, this explicit feature modeling strategy often fails in real-world scenarios because new digital forgery generation models are constantly evolving and can effectively evade the detection of traditional feature detectors. Summary of the Invention

[0003] To address the aforementioned issues, this disclosure proposes a digital multimedia forensics method and apparatus, a computing system, and a computer-readable storage medium.

[0004] According to one aspect of this disclosure, a digital multimedia forensics method is provided, the method comprising: constructing a multimodal knowledge graph based on multimodal media samples, wherein the multimodal media samples include image frames, audio signals, and text subtitles; identifying potential forgery risk points from the multimodal media samples and generating a forgery task graph targeting the potential forgery risk points; performing multimodal feature extraction on the multimodal media samples based on the forgery task graph to obtain a multi-granularity forgery clue set; structuring the multi-granularity forgery clue set into knowledge entries; performing chain-like logical reasoning based on the multimodal knowledge graph and the knowledge entries, and calculating a cross-modal consistency score to generate a reasoning tree; determining a final forgery credibility score based on the confidence level of each node in the reasoning tree; and determining whether forged content exists in the multimodal media samples based on the final forgery credibility score.

[0005] Optionally, the step of constructing a multimodal knowledge graph based on multimodal media samples includes: parsing the multimodal media samples to generate a set of knowledge entities; extracting image frame features, audio features, and text semantic features from the multimodal media samples; and constructing a multimodal knowledge graph with entities as nodes and cross-modal semantic relationships as edges based on the set of knowledge entities and the extracted image frame features, audio features, and text semantic features.

[0006] Optionally, the step of analyzing the multimodal media samples to generate a forgery task graph targeting potential forgery risk points includes: analyzing the context and style features of the multimodal media samples to identify at least one potential forgery risk point at the person, scene, or semantic level; and generating a forgery task graph containing multiple verification nodes based on the potential forgery risk points, wherein each verification node corresponds to a forgery clue category to be verified.

[0007] Optionally, based on the forged task graph, the step of performing multimodal feature extraction includes: extracting at least one of face edge artifacts, skin texture anomalies, and illumination direction differences in the multimodal media samples as a first multimodal feature; detecting at least one of inter-frame discontinuities and temporal artifacts in the multimodal media samples as a second multimodal feature; and analyzing the consistency between the semantic logic of the speech text and the visual behavior in the multimodal media samples as a third multimodal feature.

[0008] Optionally, the step of calculating the cross-modal consistency score to generate a reasoning tree includes: determining the cross-modal consistency score based on the image-speech synchronization score, the image-text semantic consistency score, and the speech-speech consistency score; and generating a reasoning tree based on the cross-modal consistency score, wherein each node of the reasoning tree corresponds to a chain of forgery evidence.

[0009] Optionally, the step of determining whether there is forged content in the multimodal media sample based on the final forgery credibility score includes: determining that there is forged content in the multimodal media sample in response to the final forgery credibility score being greater than a predetermined credibility score threshold; and determining that there is no forged content in the multimodal media sample in response to the final forgery credibility score being less than or equal to the predetermined credibility score threshold.

[0010] Optionally, the digital multimedia forensics method further includes: in response to determining that there is forged content in the multimodal media sample, generating a detection report including a forgery score, a forgery area, and evidence description.

[0011] According to another aspect of this disclosure, a digital multimedia forensics apparatus is provided, comprising: a knowledge graph construction unit configured to construct a multimodal knowledge graph based on multimodal media samples, wherein the multimodal media samples include image frames, audio signals, and text subtitles; a forgery task graph generation unit configured to identify potential forgery risk points from the multimodal media samples and generate a forgery task graph targeting the potential forgery risk points; and a feature extraction unit configured to perform multimodal feature extraction on the multimodal media samples based on the forgery task graph. The system is configured to: obtain a multi-granularity set of forgery clues; a structuring unit configured to structure the multi-granularity set of forgery clues into knowledge entries; a reasoning tree generation unit configured to perform chain-like logical reasoning based on the multimodal knowledge graph and the knowledge entries, and calculate cross-modal consistency scores to generate a reasoning tree; a credibility score determination unit configured to determine a final forgery credibility score based on the confidence level of each node in the reasoning tree; and a forgery content determination unit configured to determine whether forgery content exists in the multimodal media sample based on the final forgery credibility score.

[0012] Optionally, the knowledge graph construction unit is further configured to: parse the multimodal media samples to generate a set of knowledge entities; extract image frame features, audio features, and text semantic features from the multimodal media samples; and construct a multimodal knowledge graph with entities as nodes and cross-modal semantic relationships as edges based on the set of knowledge entities and the extracted image frame features, audio features, and text semantic features.

[0013] Optionally, the forgery task graph generation unit is further configured to: analyze the context and style features of the multimodal media sample to identify at least one potential forgery risk point at the character, scene, or semantic level; and generate a forgery task graph containing multiple verification nodes based on the potential forgery risk points, wherein each verification node corresponds to a forgery clue category to be verified.

[0014] Optionally, the feature extraction unit is further configured to: extract at least one of face edge artifacts, skin texture anomalies, and illumination direction differences in the multimodal media samples as a first multimodal feature; detect at least one of inter-frame discontinuities and temporal artifacts in the multimodal media samples as a second multimodal feature; and analyze the consistency between the semantic logic of the speech text and the visual behavior in the multimodal media samples as a third multimodal feature.

[0015] Optionally, the inference tree generation unit is further configured to: determine the cross-modal consistency score based on the image-speech synchronization score, the image-text semantic consistency score, and the speech-speech consistency score; and generate an inference tree based on the cross-modal consistency score, wherein each node of the inference tree corresponds to a chain of forgery evidence.

[0016] Optionally, the forged content determination unit is further configured to: determine that forged content exists in the multimodal media sample in response to the final forged credibility score being greater than a predetermined credibility score threshold; and determine that forged content does not exist in the multimodal media sample in response to the final forged credibility score being less than or equal to the predetermined credibility score threshold.

[0017] Optionally, the digital multimedia forensics device further includes a detection report generation unit, which is configured to generate a detection report including a forgery score, a forgery area, and evidence description in response to determining that forged content exists in the multimodal media sample.

[0018] According to another aspect of this disclosure, a computing system is provided that includes at least one computing device and at least one storage device for storing instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform the digital multimedia forensics method as described above.

[0019] According to another aspect of this disclosure, a computer-readable storage medium for storing instructions is provided, wherein when the instructions are executed by at least one computing device, the at least one computing device causes the at least one computing device to perform the digital multimedia forensics method as described above.

[0020] By adopting this disclosure, the ability of the learned implicit features to distinguish between true and false content can be effectively enhanced, thus eliminating the need to rely on a large multimodal detection model. In other words, it can achieve high-precision and robust identification of forged content. It is suitable for deployment in low-computing-power devices, edge security systems and real-time detection scenarios, and has significant advantages in computational efficiency and generalization performance. Attached Figure Description

[0021] The above and / or other objects and advantages of this disclosure will become clearer from the following description of embodiments in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart illustrating a digital multimedia forensics method according to an exemplary embodiment of the present disclosure; Figure 2 This is a diagram illustrating a fake risk planning agent according to an embodiment of the present disclosure; Figure 3 This is a diagram illustrating a multimodal forgery information mining agent according to an embodiment of the present disclosure; Figure 4 This is a diagram illustrating a multimodal reasoning agent according to an embodiment of the present disclosure; Figure 5 This is a diagram illustrating a multimodal forgery report verification agent according to an embodiment of the present disclosure; Figure 6 This is a block diagram illustrating a digital multimedia forensics apparatus according to exemplary embodiments of the present disclosure; Figure 7 This is a block diagram illustrating a computing system including at least one computing device and at least one storage device of storage instructions according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0022] The following description, in conjunction with the accompanying drawings, provides specific embodiments to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, upon understanding this disclosure, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be altered as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0023] The field of Deep Forgery Modeling (DMF) is increasingly shifting towards implicit forgery feature modeling strategies. This involves using deep models to automatically learn latent cross-modal or temporal consistency features, implicitly uncovering forgery traces, and using these implicit features to guide the differentiation between real and forged samples. However, as forgery techniques are constantly updated, implicit forgery feature modeling strategies also require incorporating more expert knowledge and retraining the model to adapt to the latest forgery techniques. Therefore, this disclosure aims to address the over-reliance on expert knowledge in implicit forgery feature modeling strategies by proposing a digital multimedia forensics method based on the prior knowledge of a multimodal large model. This method constructs a common-sense knowledge graph by mining the underlying elements of forged inputs and uses a multimodal large model as a discriminant model to identify unreasonable parts in this graph, thereby achieving the goal of identifying deep forged inputs. The advantage of this method is that it only requires the pre-trained knowledge of the large model and common-sense knowledge. Furthermore, both the graph mining and recognition processes are performed by the model. Only forged examples need to be input in advance to help the model understand different forgery styles, without the need to re-collect forged data and retrain the discriminant model. Existing digital multimedia forensics methods based on multimodal large models can be summarized into two technical approaches. The first approach utilizes context engineering, adding special prompts to enable a large model to distinguish between genuine and fake data. The second approach uses fine-tuning of a multimodal large model, adjusting the loss function to adapt it to digital multimedia forensics tasks. The former efficiently utilizes the knowledge of the large model to distinguish between genuine and fake sample pairs without retraining, but its generalization ability is limited. The latter has better generalization ability, but it heavily relies on high-quality, large-scale genuine and fake sample pairs and computational resources. This disclosure aims to propose a novel training-free digital multimedia forensics method that, instead of relying on limited context engineering, uses knowledge engineering-driven approaches, ensuring improved generalization while reducing dependence on high-computing environments.

[0024] Figure 1 This is a flowchart illustrating a digital multimedia forensics method according to an exemplary embodiment of the present disclosure.

[0025] like Figure 1 As shown, in step S101, a multimodal knowledge graph is constructed based on multimodal media samples, wherein the multimodal media samples include image frames, audio signals, and text subtitles. In this example, the multimodal media samples are parsed to generate a set of knowledge entities; image frame features, audio features, and text semantic features are extracted from the multimodal media samples; based on the set of knowledge entities and the extracted image frame features, audio features, and text semantic features, a multimodal knowledge graph is constructed with entities (including, for example, people, actions, speech content, time series, etc.) as nodes and cross-modal semantic relationships (e.g., "person A - speaking at -t1") as edges.

[0026] In step S102, potential forgery risk points are identified from the multimodal media samples, and a forgery task graph is generated targeting these potential forgery risk points. In the example, the context and style features of the multimodal media samples are analyzed to identify at least one potential forgery risk point at the person, scene, or semantic level. Based on the potential forgery risk points, a forgery task graph containing multiple verification nodes is generated, where each verification node corresponds to a forgery clue category to be verified. For example, when facial subject and voice synchronization information are detected, a task graph containing nodes such as "lip movement and voice consistency verification" and "lighting direction consistency analysis" is generated.

[0027] In step S103, based on the forgery task graph, multimodal feature extraction is performed on the multimodal media samples to obtain a multi-granularity forgery clue set. In the example, at least one of face edge artifacts, skin texture anomalies, and illumination direction differences in the multimodal media samples is extracted as the first multimodal feature; at least one of inter-frame discontinuities and temporal artifacts in the multimodal media samples is detected as the second multimodal feature; and the consistency between the semantic logic of the speech text and the visual behavior in the multimodal media samples is analyzed as the third multimodal feature. For example, low-level feature mining includes extracting face edge artifacts, skin texture anomalies, and illumination direction differences; mid-level feature mining includes detecting inter-frame discontinuities and temporal artifacts; and high-level feature mining includes analyzing whether the semantic logic of the speech text and the visual behavior are consistent.

[0028] In step S104, the multi-granularity forgery clue set is structured into knowledge entries (e.g., "(Speaker A, lip movements out of sync, time period t1-t2)" or "(Light direction, inconsistent with background, confidence level 0.85)").

[0029] In step S105, chain-like logical reasoning is performed based on the multimodal knowledge graph and knowledge entries, and cross-modal consistency scores are calculated to generate a reasoning tree. In the example, cross-modal consistency scores are determined based on image-speech synchronization scores, image-text semantic consistency scores, and speech-speech consistency scores; a reasoning tree is generated based on the cross-modal consistency scores, where each node of the reasoning tree corresponds to a chain of forgery evidence (e.g., [Step 1] Lip shape and speech are out of sync → [Step 2] Illumination direction is inconsistent → [Step 3] Facial texture artifacts are obvious → Reasoning result: high probability of forgery).

[0030] In step S106, a final forgery confidence score is determined based on the confidence level of each node in the inference tree. In step S107, based on the final forgery confidence score, it is determined whether forged content exists in the multimodal media sample. In the example, in response to the final forgery confidence score being greater than a predetermined confidence score threshold (e.g., 0.7), it is determined that forged content exists in the multimodal media sample; in response to the final forgery confidence score being less than or equal to the predetermined confidence score threshold (e.g., 0.7), it is determined that no forged content exists in the multimodal media sample. In the example, the digital multimedia forensics method further includes: in response to determining that forged content exists in the multimodal media sample, generating a detection report including a forgery score, a forgery area, and evidence description.

[0031] The digital multimedia forensics method according to exemplary embodiments of this disclosure is applicable to digital multimedia forensics scenarios with high requirements for computational efficiency and generalization ability. Existing digital multimedia forensics methods often rely on large multimodal models with a large number of parameters and high computational cost, making them difficult to apply in low-computing-power devices, real-time detection systems, or edge security scenarios. By adopting the digital multimedia forensics method according to exemplary embodiments of this disclosure, a lightweight, knowledge engineering-driven multi-agent digital multimedia forensics system can be constructed. This system consists of a planning agent, an information mining agent, an inference agent, and a verification agent working together. By enhancing the discriminativeness of the implicit forgery feature space, a large model knowledge-bridging detection process without training is achieved. This not only significantly reduces the computational complexity of the detection system and improves the overall detection efficiency, but also effectively improves the accuracy and cross-domain robustness of forged content identification in the absence of large-scale genuine and fake sample data. It can be widely applied in fields such as social media content moderation, judicial evidence collection, financial identity verification, and news media content security, and can significantly improve the efficiency and reliability of multimodal content security analysis.

[0032] Figure 2 This is a diagram illustrating a fake risk planning agent according to an embodiment of the present disclosure. Figure 3 This is a diagram illustrating a multimodal forgery information mining agent according to an embodiment of the present disclosure. Figure 4 This is a diagram illustrating a multimodal reasoning agent according to an embodiment of the present disclosure. Figure 5 This is a diagram illustrating a multimodal forgery report verification agent according to an embodiment of the present disclosure.

[0033] The digital multimedia forensics method disclosed herein comprises a workflow consisting of multiple large-scale model agents, including a planning agent, an information mining agent, a reasoning agent, and a verification agent. By simulating the thought process of human experts in distinguishing between true and false information, the four agents work collaboratively to form an automated reasoning loop based on a knowledge bridger. By analyzing input multimodal content (e.g., video frames, audio, text descriptions, etc.), the entire process from discovering forged clues and mining evidence to outputting a credible judgment is completed step by step.

[0034] The planning agent simulates the analytical thinking of human experts, performs preliminary content and style analysis on input videos or images, and automatically generates a "Forgery Risk Plan" by combining user intent and contextual information. It can determine which modalities (e.g., faces, voices, lighting, scene logic, etc.) are most likely to involve forgery behavior, and builds a forgery path reasoning framework based on knowledge graphs, which greatly reduces the reasoning search space and time.

[0035] The information mining agent formulates a multi-layered, multi-granular information mining strategy based on the forgery direction given by the planning agent. At the first level, a cross-modal knowledge graph is constructed through multi-modal feature extraction (image content, audio features, and text semantics). At the second level, a recursive information mining mechanism is used to obtain high-confidence forgery signs, such as subtle inconsistencies in facial dynamics, lip-sync errors, and background lighting mismatches. The results of the information mining agent are returned in the form of structured knowledge for subsequent logical reasoning.

[0036] Based on mined knowledge graph information, the reasoning agent utilizes the logical reasoning and common-sense inference capabilities of a large model to perform consistency analysis and multimodal comparison verification on various pieces of evidence. It progressively judges the plausibility of forgery through a chain-of-thought (CoT) mechanism and calculates cross-modal consistency scores (e.g., semantic consistency between image, speech, and text). The reasoning agent also possesses a self-correcting mechanism that automatically eliminates noisy reasoning paths, improving judgment accuracy.

[0037] This system verifies multiple judgments from the agent-fused reasoning agent, and based on graph structure similarity and semantic similarity of the knowledge graph, provides a final forgery confidence score and generates an interpretable detection report. This report identifies specific forgery areas, the causes of modal conflicts, and the corresponding knowledge reasoning paths, achieving transparency and traceability in the detection process. It achieves high-precision detection without retraining a large model, solely through knowledge engineering and multi-agent reasoning collaboration. Furthermore, it can run on devices with limited computing power and its modular agent structure allows for rapid adaptation to different detection tasks.

[0038] The following describes a more specific example. For example, given a set Given a multimodal input sample (including, for example, image frames, audio signals, and text captions), the execution process of the knowledge engineering-driven, training-free multi-agent digital multimedia forensics method according to embodiments of this disclosure includes the following steps: Step S1: Construct a multi-agent detection system based on knowledge bridging, building a workflow system consisting of four large model agents (including a planning agent, an information mining agent, an inference agent, and a verification agent). The planning agent is responsible for task analysis and forgery clue planning; the information mining agent performs multimodal feature extraction and forgery information mining; the inference agent performs logical and common-sense reasoning to generate forgery detection conclusions; and the verification agent integrates the results and outputs a detection report. Each agent can be implemented based on the large model API, and multi-agent communication and feature fusion are achieved through a unified knowledge bridging module.

[0039] Step S2: Construct a multimodal knowledge graph. For the input sample, a planning agent parses the subject, scene, and semantic relationships of the video content to generate a preliminary set of knowledge entities. Based on this, an information mining agent extracts multimodal information (including, for example, image frame features, audio features, and text semantics) to construct a multimodal knowledge graph containing nodes (entities) and edges (relationships). =( , The nodes include people, actions, speech content, time series, etc., and the edges represent cross-modal relationships (e.g., "person A - speaks at -t1").

[0040] Step S2-1: Multimodal fine-grained entity extraction (node ​​construction). For the input multimedia samples, the information mining agent drives a multimodal large model (or calls basic vision / audio tools) to perform entity recognition for different modalities, constructing a graph node set. Visual Modal Entities: By performing object detection and face recognition on video frames, "person identity," "key objects (e.g., microphone, vehicle)," "environmental scene (e.g., indoor conference room, outdoor street)," and "facial action units (e.g., blinking, opening mouth)" are extracted as visual entity nodes. Audio Modal Entities: Using automatic speech recognition and voiceprint recognition technologies, "speech segment content," "speaker's voiceprint," and "background noise type (e.g., wind sound, siren sound)" are extracted as audio entity nodes. Text Modal Entities: Through optical character recognition (OCR) to extract video subtitles and on-screen text, "key semantic words" and "timestamp information" are extracted as text entity nodes.

[0041] Step S2-2: Cross-modal semantic relationship mining (edge ​​construction). Utilizing the logical reasoning and multimodal alignment capabilities of the large model, establish connections between nodes and construct a graph edge set E. Cross-modal semantic relationships include temporal co-occurrence relationships, semantic correspondence relationships, and causal / physical logical relationships. Temporal co-occurrence relationships: If a visual entity (e.g., person A) and an audio entity (speaker A's voice) appear within the same timestamp interval, establish a "synchronous occurrence" edge. Semantic correspondence relationships: Input the extracted image scene description (e.g., "raining") and audio / text content (e.g., "bad weather today") into the large model for semantic comparison. If the two are semantically consistent, establish a "semantically consistent" edge; if there is a logical conflict (e.g., the image shows bright sunshine but the voice says it's raining), establish a "semantic conflict" edge. Causal / physical logical relationships: Analyze the physical association between visual actions and sounds, such as establishing a generative relationship of "mouth movement - generating - speech signal," or a causal relationship of "object striking - causing - impact sound."

[0042] Step S2-3: Formal Representation of the Knowledge Graph. The extracted entities and relationships are integrated to generate a structured multimodal knowledge graph. =( , Using triples to store knowledge graphs. Examples such as (person A, speaking, voice clip 1) or (scene lighting, mismatch, background shadow) provide a traversable chain of evidence for subsequent reasoning agents. Step S3: Forgery Clue Planning. The specific steps are as follows: First, analyze the context and style features of the multimodal media samples to identify at least one potential forgery risk point at the character, scene, or semantic level; based on the potential forgery risk points, generate a forgery task graph containing multiple verification nodes. Each verification node corresponds to a category of forged clues to be verified. The specific execution logic is as follows.

[0043] Step S3-1: Context-Aware Risk Point Assumptions. The planning agent first classifies the input video frame sequence and audio into scenes (e.g., news broadcasts, film clips, natural scenery, etc.). Based on a pre-built "forgery pattern knowledge base," it analyzes the spoofable attributes of the current scene. Semantic level analysis: If the scene is identified as "a famous person giving a speech," the agent marks "voice-lip alignment" and "semantic logic consistency" as high-risk points; Style level analysis: If the detected image style is "high saturation" or "anime style," the agent marks "Generative Adversarial Network (GAN) fingerprint artifacts" as risk points; Physical level analysis: If the scene contains "complex lighting changes," the agent marks "inconsistent lighting between the face and the background" as risk points.

[0044] Step S3-2: Fake mission map The instantiation and construction of the task is then performed. Based on the identified risk points, the system generates a fake task graph in the form of a directed acyclic graph using mapping rules. Node generation ( Each risk point is instantiated as a specific verification task node. Each task node contains the following attributes: Target Region of Interest (ROI): The specific coordinates or object to be detected (e.g., face region, lip region, background sky, etc.); Verification Modality: The modality combination involved (e.g., visual + audio, visual + text, etc.); Detection Operator: A description of the specific analysis instructions to be invoked (e.g., "calculate the time alignment between lip movement and phoneme", "detect pupil reflection consistency", etc.). Edge Generation ( ): Construct connection edges based on the dependencies of the detection tasks. For example, first execute the "Face Detection and Tracking" node, and its output becomes the input to the "Facial Texture Analysis" node and the "Lip Reading Consistency Verification" node, forming a cascaded detection path.

[0045] Step S4: Information Mining Phase. The information mining agent traverses the forged task graph. Based on the attributes of the task nodes, they are distributed to the corresponding mining levels to perform specific feature extraction operations. The specific steps are as follows.

[0046] Step S4-1: Task Graph Parsing and Distribution. The information mining agent analyzes the forged task graph generated in step S3. Perform topological sorting or traversal. For each verification node in the graph, the agent maps it to a low-level, mid-level, or high-level mining module based on the node's defined "verification modality" and "detection operator description".

[0047] Step S4-2: Perform hierarchical information mining. Each level of mining module executes the corresponding algorithm according to the assigned task node. Low-level feature mining (corresponding to spatial domain task nodes): For nodes in the task graph involving single-frame image quality analysis (e.g., "facial texture analysis"), the mining agent extracts pixel-level features. Specifically, this includes: using frequency domain analysis algorithms to detect edge artifacts in the face region; using statistical features to analyze abnormal smoothness of skin texture; calculating local illumination distribution vectors to quantify the difference in illumination direction between the face and the background. Mid-level feature mining (corresponding to temporal domain task nodes): For nodes in the task graph involving video sequence continuity (e.g., "inter-frame consistency detection"), the mining agent extracts temporal features. Specifically, this includes: calculating the optical flow field between adjacent frames to detect unnatural inter-frame jumps; analyzing temporal motion trajectories to identify temporal artifacts caused by video frame insertion or loss. High-level feature mining (corresponding to cross-modal semantic task nodes): For nodes in the task graph involving multimodal logical judgments (e.g., "audiovisual consistency verification"), the mining agent extracts semantic-level features. Specifically, this includes: extracting the Mel-frequency cepstral coefficients (MFCCs) of the audio and the motion trajectories of key points on the face and lips, and performing time-axis alignment analysis; extracting the text semantic vector of the speech content and the behavior description vector of the video frame, and calculating the semantic matching degree between the two.

[0048] Step S4-3: Generate a multi-granularity set of forgery clues. Each level of the mining module associates the extracted anomaly feature values ​​with the corresponding task node IDs, summarizing them to form a multi-granularity set of forgery clues. ={ 1, 2,…, }. Each clue i contains the feature type, feature value and corresponding spatiotemporal location information (e.g., "frames 15-20, lip movement amplitude and speech energy do not match").

[0049] Step S5: Forge knowledge-based clues. This involves transforming the discovered features... The knowledge entries are structured into triplets, such as "(Speaker A, lip movements out of sync, time period t1-t2)" or "(Light direction, inconsistent with background, confidence level 0.85)". All knowledge entries are injected into the knowledge bridging module for subsequent logical reasoning.

[0050] Step S6: Knowledge-Driven Reasoning. The reasoning agent receives the knowledge graph and the set of forged clues, and performs chain-of-thought reasoning through a multimodal large model to progressively determine the logical plausibility of each clue. The system defines the confidence level of each reasoning node. ∈[0,1], and make weighted judgments on unreasonable phenomena based on common sense knowledge base.

[0051] Step S7: Cross-modal consistency calculation. The inference agent calls the knowledge bridging module to calculate the multimodal consistency score.

[0052] in, The score represents the synchronization between image and speech. The score represents the semantic consistency between the image and the text. This represents the score for consistency between speech and semantics. The weights are as follows. The specific methods for determining each score and its weight are described below.

[0053] (Image-to-Speech Synchronization Score) Calculation: This score measures the degree of temporal and energy synchronization between the sound source (e.g., the speaker's mouth) in the video frame and the audio signal. The calculation method includes: first, extracting motion sequence features (e.g., lip opening / closing) of the mouth region of the face in the video; simultaneously extracting the Mel-frequency cepstral coefficients (MFCC) or energy envelope features of the audio; then, using canonical correlation analysis (CCA) and a sliding window cross-correlation function, calculating the maximum correlation coefficient between the two time series on the time axis. This coefficient, after normalization, becomes the [value / value]. .

[0054] (Image-to-Text Semantic Consistency Score) Calculation: This score measures the semantic matching degree between video content and subtitles / OCR text. The calculation method includes: using a pre-trained multimodal encoder (e.g., CLIP or BLIP) to map keyframe images into visual embedding vectors. Map the corresponding subtitle text to a text embedding vector. Calculate the cosine similarity between two vectors: .

[0055] (Speech-Semantic Consistency Score) Calculation: This score measures whether the emotion / tone of the speech is logically consistent with the text content. The calculation method includes: extracting emotional feature vectors (e.g., excitement, sadness) from the audio using a speech emotion recognition model, and simultaneously performing sentiment analysis on the transcribed text to obtain a semantic emotion vector; calculating the Euclidean distance or matching degree between the two emotion feature vectors, and then normalizing the result. .

[0056] Weight The method for determining the weights is as follows: the weights are not fixed values, but rather adaptively generated by the verification agent based on modal quality and detection confidence. The determination logic includes: first, evaluating the signal-to-noise ratio (SNR) of each modality (e.g., if a large amount of ambient noise is detected in the audio, then reducing the audio-related weights). , If the video is blurry or too dark, reduce the visual weight. The final weights must meet the normalization condition. .

[0057] Step S8: Inference Path Generation. Based on the multimodal consistency score calculated in step S7, the system performs a weighted path search in the knowledge graph to generate an inference tree structure. The specific generation steps are as follows.

[0058] Step S8-1: Abnormal Node Activation. The low consistency score calculated in step S7 (indicating high forgery risk, e.g.) is used to... <0.4) is mapped back to the corresponding entity nodes in the multimodal knowledge graph (e.g., "mouth action nodes" and "voice audio nodes"). The system marks these nodes as "high-energy anomaly nodes" as the starting anchor points of the evidence chain.

[0059] Step S8-2: Path backtracking based on causal logic. The reasoning agent starts from the "high-energy anomaly node" and performs reverse or associative searches along the edges of the graph to find the predecessor node that caused the anomaly. The system utilizes the common-sense reasoning capabilities of the large model to retain only paths that conform to logical causal relationships. Example of logical filtering rules: If "lighting anomaly" is detected, the system searches for associated "ambient light source nodes"; if there is a logical connection between the two (e.g., "indoor light sources cannot produce shadows at this angle"), the path is retained; if there is no logical connection, the path is pruned.

[0060] Step S8-3: Evidence Chain Construction and Ranking. Link the remaining valid paths to form a directed path from "lower-level features" to "higher-level fabricated conclusions," i.e., a chain of fabricated evidence. The system calculates the joint confidence of each evidence chain and selects the Top-N paths with the highest confidence to construct a reasoning tree. .

[0061] Step S9: Confidence Aggregation and Verification Phase. Verify the agent's received inference tree. It also integrates multiple sub-conclusions to calculate the final forgery credibility score. :

[0062] in, Assign importance weights to each clue. If If the value is greater than 0.7, the input sample is determined to be forged content; otherwise, it is determined that there is no forged content in the input sample.

[0063] Step S10: Result Generation and Report Output. Based on the final forgery determination result, the verification agent traces back the previous processing steps, extracts metadata, and generates a structured detection report. The specific generated items and their relationship to the preceding steps are as follows: 1. Forgery Score Generation: Directly outputs the score calculated in step S9. Value, as a quantitative indicator.

[0064] 2. Forged Region Location. Relationship Backtracking: Verify the chain of evidence with the highest contribution in the proxy backtracking reasoning tree (i.e., The system finds the "visual modality node" at the starting point of the largest chain. This is achieved by retrieving the spatial coordinates or timestamp range corresponding to the node stored in step S4 (information mining stage). For example, if the evidence chain points to "abnormal lip movement," the system retrieves the "mouth region ROI coordinates (x, y, w, h)" recorded in step S4 and marks it as a forged region.

[0065] 3. Evidence Generation. Relationship Backtracking: Extracting key path triples from the inference tree (derived from steps S5 and S8). This is achieved by having the verification agent utilize the natural language generation capabilities of the large model to transform the structured knowledge paths into readable text. Specific examples are shown below: Forgery Score: 0.92 Forgery Region: mouth area Evidence: Lip-sync mismatch with audio Reasoning Summary: Inconsistent temporal alignment detected between speech and mouth motion”. Step S11: System Optimization and Self-Correction. During long-term operation, the system automatically records the forgery confidence level of the detection results and human review feedback, updates the confidence weight distribution in the knowledge bridging module, and achieves adaptive optimization.

[0066] Figure 6 This is a block diagram illustrating a digital multimedia forensics apparatus according to an exemplary embodiment of the present disclosure.

[0067] like Figure 6As shown, a digital multimedia forensics apparatus 600 according to an exemplary embodiment of the present disclosure includes: a knowledge graph construction unit 601 configured to construct a multimodal knowledge graph based on multimodal media samples, wherein the multimodal media samples include image frames, audio signals, and text subtitles; a forgery task graph generation unit 602 configured to identify potential forgery risk points from the multimodal media samples and generate a forgery task graph targeting the potential forgery risk points; and a feature extraction unit 603 configured to perform multimodal feature extraction on the multimodal media samples based on the forgery task graph to obtain... A multi-granularity set of forgery clues is retrieved; a structuring unit 604 is configured to structure the multi-granularity set of forgery clues into knowledge entries; a reasoning tree generation unit 605 is configured to perform chain-like logical reasoning based on the multimodal knowledge graph and knowledge entries, and calculate cross-modal consistency scores to generate a reasoning tree; a credibility score determination unit 606 is configured to determine the final forgery credibility score based on the confidence of each node in the reasoning tree; and a forgery content determination unit 607 is configured to determine whether forgery content exists in the multimodal media sample based on the final forgery credibility score.

[0068] In the example, the knowledge graph construction unit 601 is further configured to: parse the multimodal media samples to generate a set of knowledge entities; extract image frame features, audio features, and text semantic features from the multimodal media samples; and construct a multimodal knowledge graph with entities as nodes and cross-modal semantic relationships as edges based on the set of knowledge entities and the extracted image frame features, audio features, and text semantic features.

[0069] In the example, the forgery task graph generation unit 602 is further configured to: analyze the context and style features of the multimodal media sample to identify at least one potential forgery risk point at the character, scene or semantic level; and generate a forgery task graph containing multiple verification nodes based on the potential forgery risk points, wherein each verification node corresponds to a forgery clue category to be verified.

[0070] In the example, the feature extraction unit 603 is further configured to: extract at least one of face edge artifacts, skin texture anomalies, and lighting direction differences in the multimodal media samples as a first multimodal feature; detect at least one of inter-frame discontinuities and temporal artifacts in the multimodal media samples as a second multimodal feature; and analyze the consistency between the semantic logic of speech text and visual behavior in the multimodal media samples as a third multimodal feature.

[0071] In the example, the inference tree generation unit 605 is further configured to: determine a cross-modal consistency score based on the image-speech synchronization score, the image-text semantic consistency score, and the speech-speech consistency score; and generate an inference tree based on the cross-modal consistency score, wherein each node of the inference tree corresponds to a chain of forgery evidence.

[0072] In the example, the forged content determination unit 607 is further configured to: determine that forged content exists in the multimodal media sample in response to the final forged credibility score being greater than a predetermined credibility score threshold; and determine that forged content does not exist in the multimodal media sample in response to the final forged credibility score being less than or equal to the predetermined credibility score threshold.

[0073] In the example, the digital multimedia forensics apparatus may also include a detection report generation unit configured to generate a detection report including a forgery score, forgery area, and evidence description in response to determining that forged content exists in a multimodal media sample.

[0074] The above combination Figures 1 to 5 The specific operations shown are respectively by Figure 6 The corresponding unit in the digital multimedia forensics apparatus 600 shown executes the procedure; specific operational details will not be elaborated here. The digital multimedia forensics apparatus according to the exemplary embodiments of this disclosure not only significantly reduces the computational complexity of the detection system and improves overall detection efficiency, but also effectively improves the accuracy and cross-domain robustness of forged content identification in the absence of large-scale genuine and fake sample data. It can be widely applied in fields such as social media content moderation, judicial evidence collection, financial identity verification, and news media content security, significantly improving the efficiency and reliability of multimodal content security analysis.

[0075] Figure 7 This is a block diagram illustrating a computing system including at least one computing device and at least one storage device of storage instructions according to an exemplary embodiment of the present disclosure.

[0076] like Figure 7 As shown, the computing system 700 provided according to an exemplary embodiment of the present invention includes a computing device 701 and a storage device 702. The storage device 702 stores computer-executable instructions. When the computer-executable instructions are executed by the computing device 701, the digital multimedia forensics method described in any of the foregoing embodiments is executed.

[0077] The computing device 701 can be deployed in a server or client, or on a node device in a distributed network environment. Furthermore, the computing device 701 can be a PC, tablet, personal digital assistant, smartphone, web application, or other device capable of executing the aforementioned set of instructions. Here, the computing device is not necessarily a single computing device; it can be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. The computing device can also be part of an integrated control system or system manager, or can be configured to interconnect locally or remotely (e.g., via wireless transmission) through an interface. In the computing device, the processor includes a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor also includes analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0078] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores instructions which, when executed by at least one computing device, cause the at least one computing device to perform the digital multimedia forensics method described in any of the foregoing embodiments. The computer-readable storage medium includes magnetic media such as floppy disks and magnetic tapes, optical media (including optical disc (CD) ROMs and DVD ROMs), magneto-optical media such as floppy discs, hardware devices such as ROMs and RAMs designed for storing and executing program commands, and flash memory. The instructions may include language code executable by a computer using an interpreter and machine language code generated by a compiler.

[0079] By adopting this disclosure, the ability of the learned implicit features to distinguish between true and false content can be effectively enhanced, thus eliminating the need to rely on a large multimodal detection model. In other words, it can achieve high-precision and robust identification of forged content. It is suitable for deployment in low-computing-power devices, edge security systems and real-time detection scenarios, and has significant advantages in computational efficiency and generalization performance.

[0080] The processes, methods, or algorithms disclosed herein can be transmitted to, or implemented by, a processing device, controller, or computer, which may include any existing programmable electronic control unit or a dedicated electronic control unit. Similarly, the processes, methods, or algorithms can be stored in various forms as data and instructions executable by a controller or computer, including but not limited to information permanently stored on non-writable storage media (such as ROM devices) and information variablely stored on writable storage media (such as floppy disks, magnetic tapes, CDs, RAM devices, and other magnetic and optical media). The processes, methods, or algorithms can also be implemented in a software executable object. Optionally, the processes, methods, or algorithms can be implemented wholly or partially using suitable hardware components (such as ASICs, FPGAs, state machines, controllers, or other hardware components or devices) or a combination of hardware components, software components, and firmware components.

[0081] Although this disclosure includes specific examples, it will be apparent to those skilled in the art that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered merely for descriptive purposes and not for limiting purposes. The description of features or aspects in each example is to be considered applicable to similar features or aspects in other examples. Suitable results may be obtained if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner and / or if components in the described system, architecture, apparatus, or circuit are replaced or supplemented with other components or their equivalents. Therefore, the scope of this disclosure is not limited by the specific embodiments but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents shall be construed as included in this disclosure.

Claims

1. A digital multimedia forensics method, characterized in that, The digital multimedia forensics method includes: A multimodal knowledge graph is constructed based on multimodal media samples, wherein the multimodal media samples include image frames, audio signals, and text subtitles; Potential forgery risk points are identified from the multimodal media samples, and a forgery task graph targeting these potential forgery risk points is generated. Based on the forgery task graph, multimodal feature extraction is performed on the multimodal media samples to obtain a multi-granularity forgery clue set; The multi-granularity forgery clue set is structured into knowledge entries; Based on the multimodal knowledge graph and the knowledge entries, chain-like logical reasoning is performed, and cross-modal consistency scores are calculated to generate a reasoning tree; Based on the confidence level of each node in the inference tree, the final forgery credibility score is determined; Based on the final forgery credibility score, it is determined whether there is forged content in the multimodal media sample.

2. The digital multimedia forensics method according to claim 1, characterized in that, The steps for constructing a multimodal knowledge graph based on multimodal media samples include: The multimodal media samples are parsed to generate a set of knowledge entities; Image frame features, audio features, and text semantic features are extracted from the multimodal media samples; Based on the knowledge entity set and the extracted image frame features, audio features and text semantic features, a multimodal knowledge graph is constructed with entities as nodes and cross-modal semantic relationships as edges.

3. The digital multimedia forensics method according to claim 1, characterized in that, The steps of analyzing the multimodal media samples to generate a forgery task graph targeting potential forgery risk points include: The contextual and stylistic features of the multimodal media samples are analyzed to identify at least one potential forgery risk point at the character, scene, or semantic level. Based on the potential forgery risk points, a forgery task graph containing multiple verification nodes is generated, wherein each verification node corresponds to a forgery clue category to be verified.

4. The digital multimedia forensics method according to claim 1, characterized in that, Based on the forged task graph, the steps for performing multimodal feature extraction include: At least one of the following features from the multimodal media samples—face edge artifacts, abnormal skin texture, and differences in illumination direction—is extracted as the first multimodal feature; Detect at least one of inter-frame discontinuity and temporal artifacts in the multimodal media samples as a second multimodal feature; The consistency between the semantic logic of speech text and visual behavior in the multimodal media samples is analyzed as a third multimodal feature.

5. The digital multimedia forensics method according to claim 1, characterized in that, The steps for calculating cross-modal consistency scores to generate a reasoning tree include: The cross-modal consistency score is determined based on the image-speech synchronization score, the image-text semantic consistency score, and the speech-speech consistency score. A reasoning tree is generated based on the cross-modal consistency score, wherein each node of the reasoning tree corresponds to a chain of forged evidence.

6. The digital multimedia forensics method according to claim 1, characterized in that, Based on the final forgery credibility score, the step of determining whether forged content exists in the multimodal media sample includes: In response to the final forgery credibility score being greater than a predetermined credibility score threshold, it is determined that forged content exists in the multimodal media sample; In response to the final forgery credibility score being less than or equal to the predetermined credibility score threshold, it is determined that there is no forged content in the multimodal media sample.

7. The digital multimedia forensics method according to claim 1, characterized in that, The digital multimedia forensics method also includes: In response to the determination that counterfeit content exists in the multimodal media sample, a detection report is generated that includes a counterfeit score, counterfeit areas, and evidence descriptions.

8. A digital multimedia forensics device, characterized in that, The digital multimedia forensics device includes: The knowledge graph construction unit is configured to construct a multimodal knowledge graph based on multimodal media samples, wherein the multimodal media samples include image frames, audio signals, and text subtitles; The forgery task graph generation unit is configured to identify potential forgery risk points from the multimodal media samples and generate a forgery task graph targeting the potential forgery risk points. The feature extraction unit is configured to perform multimodal feature extraction on the multimodal media sample based on the forgery task graph to obtain a multi-granularity forgery clue set; The structuring unit is configured to structure the multi-granularity set of forgery clues into knowledge entries; The reasoning tree generation unit is configured to perform chain-like logical reasoning based on the multimodal knowledge graph and the knowledge entries, and to calculate cross-modal consistency scores to generate a reasoning tree; The credibility score determination unit is configured to determine the final forgery credibility score based on the confidence level of each node in the inference tree; The forged content determination unit is configured to determine whether forged content exists in the multimodal media sample based on the final forged credibility score.

9. A computing system comprising at least one computing device and at least one storage device for storing instructions, characterized in that, When the instruction is executed by the at least one computing device, it causes the at least one computing device to perform the digital multimedia forensics method according to any one of claims 1 to 7.

10. A computer-readable storage medium for storing instructions, characterized in that, When the instruction is executed by at least one computing device, it causes the at least one computing device to perform the digital multimedia forensics method according to any one of claims 1 to 7.

Images