Deepfake video detection method and device based on multi-agent collaborative forensics

Abstract

The application discloses a deep fake video detection method and device based on multi-agent collaborative evidence collection, which comprises the following steps: acquiring an input video frame; observing and collecting evidence from the input video frame from different observation angles by a plurality of observation agents constructed in advance, to generate visual evidence corresponding to the observation angle of each observation agent; generating an observation report summary based on each visual evidence; inputting the observation report summary into a pre-constructed decision agent, reasoning the observation report summary by the decision agent to obtain a judgment basis; and generating a true or false judgment result based on the judgment basis. The method reconstructs the detection process into a separated architecture of multi-agent independent evidence collection and pure text decision, makes the decision chain transparent and traceable, solves the problem that the existing technology directly outputs a probability score in an end-to-end black box mode and cannot provide an auditable judgment basis, and meets the demand for result review and attribution in identity authentication and content review scenarios.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method and apparatus for detecting deepfake videos based on multi-agent collaborative forensics. Background Technology

[0002] With the rapid development of AI-powered video generation, image editing, and deepfake technologies, deepfake videos are increasingly appearing in scenarios such as identity theft, fraud attacks, and the spread of false content, posing significant security risks to internet identity authentication, financial risk control, content security auditing, and digital interaction systems. Especially against the backdrop of the continuous evolution of video generation and face editing models, deepfake face videos are constantly improving in terms of image quality, local details, temporal coherence, and overall consistency, making traditional deepfake video detection methods based on a single model or single perspective face even greater challenges.

[0003] However, existing deepfake video detection methods lack interpretability in their judgment process, making the results difficult to trace and audit. This manifests in two ways: at the result level, existing technologies typically output a true / false probability score directly in an end-to-end manner, failing to explain the specific visual cues upon which the judgment was based. Reviewers or system operators only receive a true / false label without understanding the underlying reasons. At the process level, the decision-making logic of existing technologies is embedded in the complex parameter space of deep neural networks, and the intermediate reasoning steps cannot be understood or verified by humans. Once a misjudgment occurs or an adversarial attack is launched, it is difficult to pinpoint the cause of the failure and implement targeted fixes. These shortcomings make it difficult for existing methods to meet the rigid requirements of interpretable and auditable results in scenarios such as identity authentication and content moderation. Summary of the Invention

[0004] This application provides a method and apparatus for detecting deepfake videos based on multi-agent collaborative forensics, aiming to solve the technical problem of the lack of interpretability in the discrimination process of deepfake video detection methods.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] A deepfake video detection method based on multi-agent collaborative forensics includes:

[0007] Obtain the input video frame;

[0008] Multiple pre-constructed observation agents are used to observe and collect evidence from the input video frames from different perspectives, generating visual evidence corresponding to the observation perspectives of each observation agent. During the observation and evidence collection process, each observation agent is isolated from the others and cannot see the evidence collection results of the others, and each observation agent does not output a true or false judgment result.

[0009] Based on each of the aforementioned visual evidences, a summary observation report is generated;

[0010] The observation report is summarized and input into a pre-built adjudicating agent, which then performs reasoning on the summary of observation reports to obtain a basis for judgment; wherein, the adjudicating agent does not acquire the input video frames;

[0011] Based on the aforementioned criteria, a true / false determination result is generated.

[0012] A deepfake video detection device based on multi-agent collaborative forensics includes:

[0013] The input acquisition unit is used to acquire input video frames;

[0014] The observation and evidence collection unit is used to observe and collect evidence from the input video frame from different observation perspectives by multiple pre-constructed observation agents, and generate visual evidence corresponding to the observation perspectives of each observation agent; wherein, each observation agent is isolated from each other during the observation and evidence collection process, and their evidence collection results are not visible to each other, and each observation agent does not output a true or false judgment result.

[0015] The summary generation unit is used to generate a summary observation report based on each of the aforementioned visual evidences;

[0016] The adjudication reasoning unit is used to input the observation report summary into a pre-built adjudication agent, and the adjudication agent performs reasoning on the observation report summary to obtain the judgment basis; wherein, the adjudication agent does not acquire the input video frame;

[0017] The result generation unit is used to generate a truth or falsehood determination result based on the judgment criteria.

[0018] A storage medium comprising a stored program, wherein the program is executed by a processor to perform the deepfake video detection method based on multi-agent collaborative forensics.

[0019] An electronic device includes: a processor, a memory, and a bus; the processor and the memory are connected via the bus.

[0020] The memory is used to store the program, and the processor is used to run the program, wherein the program is executed by the processor to perform the deepfake video detection method based on multi-agent collaborative forensics.

[0021] The technical solution provided in this application involves acquiring input video frames; observing and collecting evidence from different perspectives using multiple pre-built observation agents, generating visual evidence corresponding to the observation perspectives of each agent; generating a summary observation report based on the visual evidence; inputting the summary observation report into a pre-built adjudication agent, which then infers from the summary to obtain a judgment basis; and generating a authenticity determination result based on the judgment basis. This application, by reconstructing the detection process into a separate architecture of independent evidence collection by multiple agents and pure text adjudication, makes the decision-making chain transparent and traceable. It solves the technical problem of existing technologies directly outputting probability scores in an end-to-end black-box manner, failing to provide auditable judgment basis, and meets the requirements of identity authentication and content review scenarios for verifiable and attributable results. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A schematic diagram of the first process of a deepfake video detection method based on multi-agent collaborative forensics provided in this application embodiment;

[0024] Figure 2 A schematic diagram of the second process of a deepfake video detection method based on multi-agent collaborative forensics provided in an embodiment of this application;

[0025] Figure 3 A schematic diagram of the third process of a deepfake video detection method based on multi-agent collaborative forensics provided in an embodiment of this application;

[0026] Figure 4 A schematic diagram of the fourth process of a deepfake video detection method based on multi-agent collaborative forensics provided in this application embodiment;

[0027] Figure 5 A schematic diagram of the fifth process of a deepfake video detection method based on multi-agent collaborative forensics provided in this application embodiment;

[0028] Figure 6 A schematic diagram of the sixth process of a deepfake video detection method based on multi-agent collaborative forensics provided in this application embodiment;

[0029] Figure 7A schematic diagram of the seventh process of a deepfake video detection method based on multi-agent collaborative forensics provided in an embodiment of this application;

[0030] Figure 8 This is a schematic diagram of the architecture of a deepfake video detection device based on multi-agent collaborative forensics, provided in an embodiment of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0033] like Figure 1 The diagram shown is a first flowchart of a deepfake video detection method based on multi-agent collaborative forensics provided in this application embodiment, including the following steps.

[0034] S101: Obtain the input video frame.

[0035] The input video frames are a sequence of face video frames extracted from the video to be detected. The video to be detected is the video data whose authenticity needs to be determined, and its sources include, but are not limited to, user-uploaded identity authentication videos, real-time video streams collected during online interactions, or multimedia files received by content review platforms.

[0036] In some examples, the observer agent and the adjudicator agent are built on a multimodal large model. By configuring different role cue words for the multimodal large model, observer agents with different observation functions and adjudicator agents for comprehensive adjudication are derived. The multimodal large model is a multimodal large language model (MLLM) with visual understanding and text generation capabilities.

[0037] It should be noted that multimodal large language models can process both image and text inputs simultaneously and generate text-based output, making them suitable as the base model for various intelligent agents. By configuring different system prompts and role prompts for the multimodal large model, it is possible to guide it to exhibit different behavioral patterns in different task scenarios, thereby deriving multiple intelligent agents with different perspectives and output formats.

[0038] S102: By using multiple pre-built observation agents, the input video frames are observed and evidence is collected from different observation perspectives, generating visual evidence corresponding to the observation perspectives of each observation agent.

[0039] In this system, each observation agent is isolated from the others during the observation and evidence collection process, and their respective evidence collection results are not visible to each other. Furthermore, each observation agent does not output a veracity determination result. Isolation means that each observation agent is independent at both the data and communication levels; that is, each observation agent can only obtain the input video frames and corresponding role prompts during the observation and evidence collection process, and cannot obtain the evidence collection results and role prompts of other observation agents.

[0040] Optionally, the multiple observation agents include a texture detail observation agent, an illumination consistency observation agent, a temporal motion observation agent, and a physical plausibility observation agent; wherein, the texture detail observation agent, the illumination consistency observation agent, the temporal motion observation agent, and the physical plausibility observation agent are constructed through corresponding role prompts, and each observation agent determines its corresponding observation perspective based on the corresponding role prompts.

[0041] It should be noted that each observation agent is pre-built using corresponding role prompts. These prompts assign different observation perspectives and evidence-gathering functions to each agent. The role prompts are text-based instructions that include descriptions of the observation target, definitions of the observation scope, and requirements for the output format. By configuring different role prompts for different observation agents, each agent can focus on different dimensions of forgery traces, thereby achieving parallel evidence gathering from multiple perspectives.

[0042] In a possible implementation, role prompts include system prompts and role prompts. System prompts define the basic roles and behavioral norms of the multimodal large model, while role prompts specify the specific observation perspective and evidence output requirements of the observing agent. For example, for a texture detail observing agent, the system prompt could be set to "You are a professional deepfake video forensics expert," and the role prompt could be set to "Please observe and collect evidence from the perspective of texture details in the input video frame, paying attention to skin texture anomalies and edge transition anomalies, and output visual evidence in a structured text format."

[0043] Optionally, the process of generating visual evidence corresponding to the observation perspectives of each observation agent by pre-constructing multiple observation agents to observe and collect evidence from the input video frames from different perspectives can be found in [reference needed]. Figure 2 The steps are shown.

[0044] Optionally, for observing the training process of the agent, see [link to relevant documentation]. Figure 5 The steps are shown.

[0045] S103: Generate a summary of observation reports based on various visual evidences.

[0046] Visual evidence consists of textual descriptions output by each observing agent after observing and collecting evidence from the input video frames based on their respective viewing perspectives. These descriptions characterize anomalous information associated with the corresponding viewing perspective. The observation reports are compiled as plain text data and do not include image or video data of the input video frames.

[0047] It should be noted that the observation report summary is composed of the text content of each visual piece of evidence arranged in a preset splicing order. This preset splicing order is determined based on the preset arrangement order of the observation perspectives of each observation agent. The preset splicing order is used to ensure that the arrangement position of the visual evidence output by each observation agent in the observation report summary is fixed, thereby enabling the adjudicating agent to read and parse the evidence information from each observation perspective in a stable order, avoiding the inability of the adjudicating agent to accurately identify the source perspective of each piece of evidence due to changes in the splicing order.

[0048] In a possible implementation, the preset splicing order is as follows: visual evidence of texture details, visual evidence of lighting consistency, visual evidence of temporal motion, and visual evidence of physical plausibility are arranged in sequence.

[0049] Optionally, the implementation process for generating a summary observation report based on various visual pieces of evidence can be found in [link to documentation]. Figure 3 The steps are shown.

[0050] S104: Input the summary of observation reports into the pre-built adjudication agent, and use the adjudication agent to reason about the summary of observation reports to obtain the basis for judgment.

[0051] The adjudicating agent does not acquire input video frames. The adjudicating agent is built upon a multimodal large model, differing from the observation agent in that the observation agent is configured to extract visual evidence from a specific perspective, while the adjudicating agent is configured to synthesize various visual evidence for reasoning and judgment. The fact that the adjudicating agent does not acquire input video frames means that its input data only includes a summary of observation reports and does not include image data, video data, or feature vector data of the input video frames.

[0052] It should be noted that the judgment criterion is textual information describing the reasoning process and conclusion of the adjudicating agent in synthesizing various visual evidences. The judgment criterion records the various visual pieces of evidence relied upon by the adjudicating agent during the reasoning process, the identification results of the relationships and contradictions between these pieces of evidence, and the logical process of deriving the final conclusion based on these identification results. By outputting the judgment criterion, the final truth or falsehood determination result is accompanied by a traceable chain of reasoning.

[0053] Optionally, the judgment criteria and the results of the authenticity determination can be output according to a preset output format.

[0054] In a possible implementation, the criteria for judgment are written into the first tag, and the result of the authenticity judgment is written into the second tag.

[0055] For example, the first label is <explanation>The tag, the second tag is <answer>The authenticity of the label is determined in <answer>The label indicates whether it is real or fake.

[0056] Optionally, the observation report summary can be input into a pre-built adjudication agent. The adjudication agent then performs reasoning on the observation report summary to obtain the basis for judgment. For details on this process, please refer to [link to relevant documentation]. Figure 4 The steps are shown.

[0057] Optionally, the training process for the adjudicating agent can be found in [reference needed]. Figure 5 The steps are shown.

[0058] S105: Generate a true / false judgment result based on the judgment criteria.

[0059] The authenticity determination result is a binary classification result used to characterize whether the input video frame belongs to a real video or a fake video. The authenticity determination result is generated based on the judgment criteria, that is, after the adjudicating agent completes the reasoning of the summary of observation reports and obtains the judgment criteria, it determines the final judgment category according to the judgment criteria.

[0060] It should be noted that the generation process of the truth / false determination result and the judgment basis are sequential: the adjudicating agent first performs textual semantic understanding and logical reasoning on the summary of observation reports to obtain the judgment basis used to describe the reasoning process and conclusion, and then outputs the truth / false determination result based on the judgment basis. This two-level output mechanism ensures that the final truth / false determination result comes with a complete reasoning chain, so that users can not only obtain the truth / false conclusion, but also trace the evidence and reasoning process on which the conclusion is based.

[0061] The process described in S101-S105 above involves multiple observation agents performing blind observation and evidence collection on the input video frames in parallel from different perspectives. The adjudication agent then performs independent reasoning based on the observation reports compiled from plain text. This breaks down the decision chain of the detection process into multiple independently inspectable links, with each link outputting traceable and verifiable results. Ultimately, a truth or falsehood judgment result with accompanying reasoning is obtained, thereby significantly improving the interpretability of the detection process and the auditability of the results while ensuring the accuracy of the detection.

[0062] like Figure 2 The diagram shown is a second flowchart of a deepfake video detection method based on multi-agent collaborative forensics provided in this application embodiment, including the following steps.

[0063] S201: The intelligent agent observes the texture details and collects evidence from the perspective of texture detail observation, generating visual evidence of texture details.

[0064] Among them, visual evidence of texture details is used to characterize skin texture anomalies and edge transition anomalies in the input video frames. Skin texture anomalies include abnormalities in the uniformity of skin surface texture distribution, abnormal texture frequency, abnormal pore morphology, and abnormal wrinkle direction; edge transition anomalies include abnormalities in the sharpness of the edge transition region between the face region and the background region, abnormalities in the smoothness of the edge shape, and abnormal gradient changes in pixel values ​​on both sides of the edge.

[0065] In some examples, the texture detail observation agent performs fine-grained analysis of the facial region in the input video frame based on corresponding role cues. It focuses on the texture distribution of the skin surface, pore morphology, wrinkle direction, the sharpness and naturalness of the edge transition between the skin and the background, and texture anomalies in fine-grained areas such as the eyes and mouth. Forged face videos typically exhibit abnormal traces in these areas, such as overly smooth textures, harsh edge transitions, missing details, or texture repetition.

[0066] It's important to note that facial regions in deepfake videos are typically synthesized using generative models or modified from the original facial region by generative models. When synthesizing facial textures, generative models often have limited ability to model real skin textures, failing to generate details that perfectly match real skin textures. This results in anomalies such as overly smoothed textures, texture repetition, or missing texture details in the synthesized region. Furthermore, when stitching the synthesized face to the original background, differences in texture distribution between the generated and original regions can easily lead to abrupt and unnatural edge transitions at the stitching boundaries. Therefore, analyzing the texture details and edge transition features of the facial region can effectively identify forgery traces in deepfake videos.

[0067] In a possible implementation, the visual evidence of texture details adopts a structured text data format, including evidence type identifier, evidence description information, and evidence location information. For example, the evidence type identifier is "texture detail abnormality", the evidence description information is "the skin texture in the area below the left eye is unnaturally smooth and lacks the texture undulations that real skin should have", and the evidence location information is "the coordinates of the face region in the input video frame (x1, y1, x2, y2)".

[0068] S202: By observing the intelligent agent through illumination consistency, observe and collect evidence from the perspective of illumination consistency of the input video frame, and generate visual evidence of illumination consistency.

[0069] Among them, visual evidence of illumination consistency is used to characterize light source direction anomalies, shadow anomalies, and highlight anomalies in the input video frame. Light source direction anomalies include anomalies such as inconsistent light source directions reflected in different areas of the face; shadow anomalies include anomalies such as the projection direction of facial shadows not matching the preset light source direction and shadow shape distortion; highlight anomalies include anomalies such as positional shifts in highlight areas and inconsistent distribution of highlight intensity.

[0070] In some examples, the illumination consistency observation agent performs illumination analysis on the facial regions in the input video frames based on corresponding role prompts. It focuses on whether the direction of the light source on the facial surface, the consistency of the illumination intensity distribution, the projection direction and shape of shadows in different facial areas, the brightness and position of highlight areas match the preset light source direction, and whether the illumination transition at the edges of the facial contours is natural. Forged face videos typically exhibit abnormalities in different facial areas, such as inconsistent light source directions, distorted shadow shapes, shifted highlight positions, or inconsistent illumination intensity distribution.

[0071] It's important to note that in real face videos, the lighting conditions are consistent across all areas of the face within the same scene. This means that the direction of the light source should be the same across all areas of the face, the direction of shadows should be consistent, and the distribution of highlights should conform to the physical relationship between the light source direction and the facial geometry. However, in deepfake videos, when the lighting conditions of the synthesized face differ from the original background, or when different areas of the face are synthesized using different generation conditions, anomalies such as conflicting light source directions, distorted shadow shapes, and shifted highlight positions may occur. Therefore, analyzing the lighting consistency characteristics of face regions can effectively identify lighting anomalies in deepfake videos.

[0072] In a possible implementation, the visual evidence of illumination consistency adopts a structured text data format, including evidence type identifier, evidence description information, and evidence location information. For example, the evidence type identifier is "illumination consistency anomaly", the evidence description information is "the highlight direction on the left side of the bridge of the nose is inconsistent with the highlight direction on the right cheekbone, indicating a conflict in the light source direction of different areas of the face", and the evidence location information is "the coordinates of the face region (x1, y1, x2, y2) in the input video frame".

[0073] S203: Observe the intelligent agent through temporal motion, observe and collect evidence from the perspective of temporal motion, and generate temporal motion visual evidence.

[0074] Temporal motion visual evidence is used to characterize inter-frame motion coherence anomalies and facial dynamic naturalness anomalies in the input video frames. Inter-frame motion coherence anomalies include abrupt changes in the motion trajectory of facial key points between adjacent video frames, abrupt changes in motion direction, and discontinuous motion speed; facial dynamic naturalness anomalies include rhythmic anomalies in facial expression changes, abnormal blinking frequency, and asynchrony between mouth opening and closing movements and speech.

[0075] In some examples, the temporal motion observation agent performs temporal analysis on input video frames based on corresponding role cues. It focuses on the continuity of facial keypoint motion trajectories between adjacent video frames, the naturalness of facial expression changes, the temporal regularity of dynamic behaviors such as blinking and mouth opening and closing, and the coordination between head movements and facial expression changes. Forged face videos typically exhibit abnormal traces in inter-frame motion, such as keypoint jumps, inconsistent motion trajectories, abrupt expression changes, or decoupling between head movements and facial expressions.

[0076] It's important to note that in real face videos, facial movement is continuous and smooth in time. Changes in the positions of facial key points between adjacent video frames should be gradual, and the motion trajectory should conform to the laws of physical motion. However, in deepfake videos, due to the limited ability of generative models to model temporal dependencies, or the lack of constraints on motion coherence during frame generation, key point positions often exhibit discontinuous jumps between adjacent video frames, resulting in a lack of smoothness in the motion trajectory. Furthermore, when constructing fake face videos using a frame-by-frame generation method, the independent generation processes of each frame can lead to a lack of temporal regularity in facial expression changes, resulting in abrupt changes in expressions and unnatural dynamic behaviors. Therefore, analyzing the temporal characteristics of facial motion trajectories and dynamic behaviors in video frame sequences can effectively identify temporal anomalies in deepfake videos.

[0077] In a possible implementation, the temporal motion visual evidence adopts a structured text data format, including evidence type identifier, evidence description information, and evidence location information. For example, the evidence type identifier is "temporal motion anomaly", the evidence description information is "a non-continuous jump occurs at the left eye key point position between frame 3 and frame 7, and the real face motion trajectory should be a smooth transition", and the evidence location information is "frames 3 to 7 in the input video frame sequence".

[0078] S204: By observing the intelligent agent through physical rationality, observe and collect evidence from the perspective of physical rationality observation of the input video frame, and generate visual evidence of physical rationality.

[0079] Among them, visual evidence of physical plausibility is used to characterize geometric structural anomalies and physical law anomalies in the input video frames. Geometric structural anomalies include abnormal geometric consistency of the face region, abnormal spatial proportions between facial features, and abnormal facial symmetry; physical law anomalies include abnormal mapping between face pose and head pose, abnormal geometric continuity between face edges and background, and abnormal projection relationships.

[0080] In some examples, the physical plausibility observation agent analyzes the physical laws of input video frames based on corresponding role prompts, focusing on the geometric consistency of the face region, the reasonableness of the spatial proportions between facial features, whether the mapping relationship between face pose and head pose conforms to physical constraints, and the geometric continuity between face edges and the background. Forged face videos typically exhibit abnormalities such as disproportionate facial features, pose mapping relationships violating physical constraints, and obvious stitching marks between face edges and the background.

[0081] It's important to note that the 3D geometry of a real human face follows defined physical laws. For example, the spatial proportions between facial features should be within a reasonable range, facial symmetry should meet biological requirements, and the mapping between facial and head poses should conform to the laws of rigid body motion. In deepfake videos, when generative models lack the ability to model 3D geometry, or when the synthesized face's 3D structure differs from the original face, anomalies such as disproportionate facial features and pose mappings violating physical constraints may occur. Furthermore, when the synthesized face region is stitched together with the original background region, the geometric continuity between the two regions is disrupted, potentially resulting in noticeable stitching marks at the edges of the face where it meets the background. Therefore, analyzing the degree to which the geometric structure and physical laws of the face region conform can effectively identify geometric anomalies in deepfake videos.

[0082] In a possible implementation, the visual evidence of physical plausibility adopts a structured text data format, including evidence type identifier, evidence description information, and evidence location information. For example, the evidence type identifier is "physical plausibility anomaly", the evidence description information is "the proportion of the triangular structure formed by the tip of the nose and the line connecting the left and right eyes is out of proportion and does not conform to the three-dimensional geometric constraints of a real human face", and the evidence location information is "the coordinates of the face region (x1, y1, x2, y2) in the input video frame".

[0083] The process described in S201-S204 above involves four different observation agents—texture detail observation agent, illumination consistency observation agent, temporal motion observation agent, and physical plausibility observation agent—collecting evidence in parallel from the same set of input video frames. Each observation agent is isolated from the others and their evidence collection results are not visible to each other, ensuring the independence and diversity of each visual piece of evidence, avoiding evidence convergence, and providing a comprehensive and unbiased evidentiary basis for subsequent adjudication.

[0084] like Figure 3 The diagram shown is a third flowchart of a deepfake video detection method based on multi-agent collaborative forensics provided in this application embodiment, including the following steps.

[0085] S301: Obtain visual evidence output by each observing agent.

[0086] The visual evidence is structured text data; the structured text data includes evidence type identifiers, evidence description information, and evidence location information. The evidence type identifier is used to characterize the type of anomaly to which the visual evidence belongs, such as "texture detail anomaly," "illumination consistency anomaly," "temporal motion anomaly," and "physical plausibility anomaly." The evidence description information is used to provide a textual description of the observed anomalous traces. The evidence location information is used to indicate the spatial and / or temporal location of the anomalous traces in the input video frame or video frame sequence.

[0087] In some examples, visual evidence output by each observation agent is obtained, including: reading the textual evidence records generated by each observation agent at a preset output position, wherein the textual evidence records contain evidence type identifiers, evidence description information and evidence location information.

[0088] In a possible implementation, the visual evidence output by each observing agent is encapsulated in a preset label format.

[0089] For example, the texture detail observation agent outputs visual evidence of texture details and writes it into... <observation>In the label, the illumination consistency visual evidence output by the illumination consistency observation agent is written into... <observation>The tag includes the temporal motion visual evidence output by the temporal motion observation agent. <observation>The label includes the visual evidence of physical plausibility output by the physical plausibility observation agent. <observation>In the tag.

[0090] It should be noted that using a structured text data format to output visual evidence ensures that the evidence collection results output by various observation agents have a unified format specification, facilitating subsequent splicing processing and automatic parsing in the adjudication stage. Evidence type identification enables the adjudication agent to quickly identify the source perspective and anomaly type of each piece of evidence, evidence description information provides a specific description of the anomaly traces, and evidence location information indicates the spatiotemporal location of the anomaly traces. These three elements together constitute a complete and parsable evidence collection record.

[0091] S302: According to the preset splicing order, splice together the various visual evidences and generate a summary observation report.

[0092] The observation reports are compiled as plain text data. They do not include image data, video data, or feature vector data from the input video frames; they consist solely of textualized visual evidence output by each observation agent.

[0093] In some examples, visual evidence is spliced ​​together according to a preset splicing order to generate a summary observation report. This includes: splicing the text content of each visual evidence in sequence according to the order of visual evidence of texture detail, visual evidence of lighting consistency, visual evidence of temporal motion, and visual evidence of physical plausibility, and adding separators between adjacent visual evidence to generate a summary observation report.

[0094] It should be noted that the observation report is compiled as plain text data and does not include image data, video data, or feature vector data of the input video frames. The content of the observation report consists only of textualized visual evidence output by each observation agent, stripping away the original image and video information, so that subsequent adjudication can only be based on textual evidence for reasoning.

[0095] The process described in S301-S302 above splices the structured visual evidence output by each observation agent into a plain text observation report summary. This summary strips away the original image and video data, preventing the adjudicating agent from directly obtaining the pixel information of the input video frame. This cuts off the visual bias in the adjudication process from the source of input, ensuring the objectivity and fairness of the adjudication process.

[0096] like Figure 4 The diagram shown is a fourth flowchart of a deepfake video detection method based on multi-agent collaborative forensics provided in this application embodiment, including the following steps.

[0097] S401: Input the observation report summary into the pre-built adjudication agent. The adjudication agent performs text semantic understanding and logical reasoning on the observation report summary to identify the correlation and contradiction between various visual evidences.

[0098] Among these, textual semantic understanding involves the adjudicating agent performing semantic analysis on the text content in the summary of observation reports, including identifying the anomaly type, location, and degree described by each piece of visual evidence, and extracting key information elements from each piece of evidence. Logical reasoning involves the adjudicating agent performing comprehensive reasoning based on the logical relationships between the various pieces of visual evidence, including identifying the correlations and contradictions between the pieces of evidence.

[0099] It should be noted that correlation refers to the mutual corroboration between visual evidence from different observation perspectives in terms of anomaly type, location, or cause; contradiction refers to the conflict between visual evidence from different observation perspectives in terms of anomaly manifestation or cause. For example, when a texture detail observation agent detects a skin texture anomaly in a certain area, and a lighting consistency observation agent detects a lighting anomaly in the same area, there is a spatiotemporal correlation between these two pieces of evidence, mutually corroborating the possibility of forgery in that area. Conversely, when a texture detail observation agent detects an anomaly of excessively smooth skin texture, but a lighting consistency observation agent does not detect any lighting anomaly, there may be a contradiction between these two pieces of evidence, requiring a decision agent to further determine which piece of evidence is more credible.

[0100] In a possible implementation, the adjudicating agent comprehensively analyzes the evidence type identifiers, evidence description information, and evidence location information of each structured visual evidence in the observation report summary to identify the spatiotemporal correlation between evidence with the same or similar evidence location information, and to determine the logical relationship of whether the abnormal traces described by each piece of evidence point to the same source of forgery.

[0101] S402: Generate judgment criteria based on correlation and contradiction relationships.

[0102] The judgment criteria consist of textual information describing the reasoning process and conclusions of the adjudicating agent in synthesizing various visual evidences. The judgment criteria include the following elements: a summary of each piece of visual evidence, a description of the relationships and contradictions between the various pieces of visual evidence, the adjudicating agent's assessment of the credibility of each piece of visual evidence, and a description of the reasoning process used to arrive at the final conclusion based on the above information.

[0103] The process described in S401-S402 involves the adjudicating agent performing textual semantic understanding and logical reasoning on the summary of observation reports to identify the relationships and contradictions between various visual pieces of evidence, and generating a judgment basis based on the identification results. This judgment basis fully records the reasoning process and conclusions of the adjudicating agent in synthesizing various visual evidence, ensuring that the final authenticity determination result is accompanied by a traceable judgment basis, thus meeting the requirements for interpretable and auditable results in scenarios such as identity authentication and content review.

[0104] like Figure 5 The diagram shown is a fifth flowchart of a deepfake video detection method based on multi-agent collaborative forensics provided in this application embodiment, including the following steps.

[0105] S501: Obtain sample video frames and their authenticity type.

[0106] The sample video frames are labeled video data used to train the observation agent and the adjudication agent. The authenticity type is a manually labeled tag for the sample video frame, indicating whether the sample video frame is a real video or a fake video, and if it is a fake video, the corresponding forgery method.

[0107] In some examples, the authenticity type includes a real type and various fake types. The real type indicates that the sample video frame is an unfake real face video; fake types include, but are not limited to, face swap, face reenactment, and fully fake face. Face swap (Fs) refers to replacing the source face with the identity information of the target face, but retaining the target face's expression and pose; face reenactment (Fr) refers to driving the source face to perform the target face's expression and head pose; fully fake face (EFS) refers to a face video completely generated from scratch by a generative model.

[0108] S502: Based on the authenticity type of sample video frames, combined with a pre-built vocabulary, construct supervisory labels.

[0109] The pre-built lexicon is a pre-constructed lexicon used to store descriptive words corresponding to each observation perspective. The supervision labels are the label data used as training targets during the supervision fine-tuning stage, including visual evidence supervision labels corresponding to each observation agent and judgment basis supervision labels and authenticity judgment results supervision labels corresponding to each adjudication agent.

[0110] In some examples, the pre-defined vocabulary includes a first set of positive descriptors and a first set of negative descriptors corresponding to the texture detail observation viewpoint; a second set of positive descriptors and a second set of negative descriptors corresponding to the illumination consistency observation viewpoint; a third set of positive descriptors and a third set of negative descriptors corresponding to the temporal motion observation viewpoint; and a fourth set of positive descriptors and a fourth set of negative descriptors corresponding to the physical plausibility observation viewpoint. The positive descriptors in each set of positive descriptors are used to characterize the natural, harmonious, and consistent visual appearance of a real human face under the corresponding observation viewpoint, while the negative descriptors in each set of negative descriptors are used to characterize any abnormal traces that may appear on a fake face under the corresponding observation viewpoint.

[0111] For example, the first set of positive descriptive terms includes terms such as "clear and natural skin texture" and "smooth edge transition," while the first set of negative descriptive terms includes terms such as "overly smooth skin texture" and "harsh edge transition." The second set of positive descriptive terms includes terms such as "consistent light source direction" and "harmonious shadow distribution," while the second set of negative descriptive terms includes terms such as "conflicting light source directions" and "distorted shadow shape." The third set of positive descriptive terms includes terms such as "coherent motion between frames" and "natural facial dynamics," while the third set of negative descriptive terms includes terms such as "abrupt motion between frames" and "stiff facial expressions." The fourth set of positive descriptive terms includes terms such as "reasonable geometric structure" and "conforms to physical constraints," while the fourth set of negative descriptive terms includes terms such as "distorted geometric structure" and "violates physical laws."

[0112] Optionally, the implementation process of constructing supervised labels based on the authenticity type of sample video frames and a pre-built lexicon can be found in [link to documentation]. Figure 6 The steps are shown.

[0113] S503: Based on supervised labels, supervised fine-tuning is performed on each observation agent and adjudicator agent to ensure that each observation agent stably outputs the corresponding visual evidence and the adjudicator agent stably outputs the judgment basis and the truth or falsehood judgment result.

[0114] Supervised fine-tuning (SFT) is a training method that adjusts the parameters of a pre-trained model based on labeled data. In supervised fine-tuning, the supervision labels are used as the training target. The difference between the model output and the supervision labels is calculated, and the model parameters are updated based on this difference, so that the distribution of the model output gradually approximates the target distribution indicated by the supervision labels.

[0115] It's important to note that the goal of supervised fine-tuning is not to directly train the observation and adjudication agents to distinguish between genuine and fake videos, but rather to enable the observation agents to consistently adhere to their respective role assignments and output formats, and the adjudication agents to consistently output judgment criteria and authenticity determinations, ensuring that the outputs of each agent can be reliably parsed. During the supervised fine-tuning phase, each agent learns through a large number of labeled samples how to observe and collect evidence and make adjudicative inferences according to the requirements of role prompts, and how to output visual evidence, judgment criteria, and authenticity determinations in a standardized, structured text format. After the supervised fine-tuning phase is completed, the output formats of each agent will tend to stabilize, thus providing a stable output space for the subsequent reinforcement learning fine-tuning phase.

[0116] In a possible implementation, based on the supervision label, the network parameters of each observation agent and the adjudicator agent are fine-tuned by supervision, including: using the supervision label as the supervision signal, calculating the difference between the visual evidence output by each observation agent and the corresponding supervision label, and the difference between the judgment basis and the authenticity judgment result output by the adjudicator agent and the corresponding supervision label; and updating the network parameters of each observation agent and the adjudicator agent based on the difference.

[0117] Optionally, the implementation process for supervised fine-tuning of each observer agent and adjudicator agent based on supervised labels can be found in [link to documentation]. Figure 7 The steps are shown.

[0118] S504: Fine-tuning of each observer agent and adjudicator agent through reinforcement learning using a group relative policy optimization algorithm.

[0119] The reward function for fine-tuning reinforcement learning includes length reward and format reward. The length reward is used to constrain the length of the visual evidence output by each observer agent and the length of the judgment basis output by the adjudicator agent. The format reward is used to constrain each observer agent and adjudicator agent to output according to a preset output format.

[0120] It should be noted that Group Relative Policy Optimization (GRPO) is a reinforcement learning policy optimization method that updates the policy by sampling multiple candidate outputs for the same input and estimating the relative advantage within each group. Compared to traditional policy gradient methods, GRPO reduces the variance of policy gradient estimation through relative comparison within groups, thereby improving the stability and efficiency of training.

[0121] In some examples, reinforcement learning fine-tuning is performed on each observer agent and the adjudicator agent through a group relative policy optimization algorithm, including: sampling multiple candidate outputs for each observer agent and the adjudicator agent for the same sample video frame; calculating the policy gradient based on the relative advantage estimation of each candidate output within the group; and updating the policy parameters of each observer agent and the adjudicator agent based on the policy gradient.

[0122] In a possible implementation, a length reward is used to constrain the text length of visual evidence output by each observation agent and the text length of judgment criteria output by the adjudicating agent to be within a preset length range. A format reward is used to constrain each observation agent and adjudicating agent to output according to a preset output format, which includes a label format and a content structure format.

[0123] In a possible implementation, length rewards and format rewards are shared by all observation agents and adjudicating agents. By setting length rewards, invalid text that is too long or too short can be avoided from being output by each agent; by setting format rewards, it can be ensured that each agent maintains the standardization of its output format during the reinforcement learning fine-tuning phase.

[0124] In some examples, the reward function fine-tuned by reinforcement learning also includes a discriminative reward. The discriminative reward determines the reward value based on the degree of matching between the authenticity type of a sample video frame and the authenticity determination result. The discriminative reward can employ an asymmetric design, where the penalty coefficient for a false negative is greater than the penalty coefficient for a false positive. A false negative occurs when the authenticity type of the sample video frame is fake and the authenticity determination result is real; a false positive occurs when the authenticity type of the sample video frame is real and the authenticity determination result is fake.

[0125] It's important to note that in practical applications of deepfake video detection, the harm caused by false negatives (mistaking fake videos for real ones) far outweighs that of false positives (mistaking real videos for fakes). False negatives mean that fake videos pass the detection system, potentially leading to serious consequences such as identity theft and fraudulent attacks; while false positives, although affecting user experience, do not pose a security risk. Therefore, during the reinforcement learning fine-tuning phase, applying a higher penalty coefficient to false negatives than to false positives can guide the agents to be more cautious in their judgments during training, effectively reducing the risk of missing fake video detection and improving the reliability of the detection system in high-security scenarios.

[0126] The process described in S501-S504 above ensures that each agent stably adheres to its role and output format through a supervised fine-tuning phase, and trains the anti-counterfeiting ability of each agent through a reinforcement learning fine-tuning phase. The training objectives of the two phases have different focuses and are progressively advanced, which not only ensures the structural stability and parsability of the output of each agent, but also ensures the accuracy of detection, avoiding the problem of mutual interference between unstable output format and anti-counterfeiting ability training in end-to-end training.

[0127] like Figure 6 The diagram shown is a sixth flowchart of a deepfake video detection method based on multi-agent collaborative forensics provided in this application embodiment, including the following steps.

[0128] S601: Select target descriptive words from a pre-set dictionary based on the authenticity type of the sample video frames.

[0129] The types of authenticity include real faces, face swaps, face reenactments, and fully synthetic faces; the pre-built vocabulary includes a set of positive descriptive words and a set of negative descriptive words; the positive descriptive words in the set of positive descriptive words are used to describe the natural and coordinated visual appearance of real faces from various viewing angles; the negative descriptive words in the set of negative descriptive words are used to describe the abnormal traces that appear on fake faces from various viewing angles.

[0130] Optionally, the positive and negative descriptor sets correspond to various observation perspectives. Specifically, positive descriptors corresponding to the texture detail observation perspective are used to describe the natural appearance of a real face in terms of texture detail, while negative descriptors are used to describe abnormal traces in the texture detail of a fake face; positive descriptors corresponding to the illumination consistency observation perspective are used to describe the harmonious appearance of a real face in terms of illumination consistency, while negative descriptors are used to describe abnormal traces in the illumination consistency of a fake face; positive descriptors corresponding to the temporal motion observation perspective are used to describe the coherent appearance of a real face in terms of temporal motion, while negative descriptors are used to describe abnormal traces in the temporal motion of a fake face; and positive descriptors corresponding to the physical plausibility observation perspective are used to describe the plausible appearance of a real face in terms of physical plausibility, while negative descriptors are used to describe abnormal traces in the physical plausibility of a fake face.

[0131] It should be noted that the pre-built lexicon is constructed based on the following principle: for each observation perspective, a set of positive descriptive terms and a set of negative descriptive terms are constructed separately. This ensures that the positive and negative descriptive terms maintain consistency in descriptive style, granularity, and sentence structure, differing only in the content's positive or negative connotation. This construction method guarantees that the supervision labels generated under different truth / falsehood types have a unified style and format, enabling each observation agent to learn a stable output format during the supervision fine-tuning phase.

[0132] S602: Generate supervisory labels corresponding to the observation perspectives of each observing agent based on the target descriptor.

[0133] The supervision labels include visual evidence supervision labels for each observation agent, judgment basis supervision labels for the adjudicating agent, and authenticity judgment result supervision labels. The visual evidence supervision labels for each observation agent serve as training targets for the outputs of each observation agent during the supervision fine-tuning phase; the judgment basis supervision labels for the adjudicating agent serve as training targets for the judgment basis outputs of the adjudicating agent during the supervision fine-tuning phase; and the authenticity judgment result supervision labels for the adjudicating agent serve as training targets for the authenticity judgment results output by the adjudicating agent during the supervision fine-tuning phase.

[0134] In a possible implementation, based on the target descriptor, supervision labels corresponding to the observation perspectives of each observation agent are generated, including: determining whether the type of the target descriptor corresponding to each observation perspective is a positive descriptor or a negative descriptor based on the authenticity type of the sample video frame; if the authenticity type is real, the target descriptor corresponding to each observation perspective is selected from the corresponding set of positive descriptors; if the authenticity type is face swapping, face replay, or fully synthetic face, the target descriptor corresponding to each observation perspective is selected from the corresponding set of negative descriptors; and based on the selected target descriptors corresponding to each observation perspective, the supervision labels of each observation agent and the supervision labels of the adjudicating agent are filled in.

[0135] For example, if the authenticity type of the sample video frame is face swapping, then "skin texture is too smooth" is selected from the negative descriptor set corresponding to the texture detail observation perspective, "light source direction conflict" is selected from the negative descriptor set corresponding to the illumination consistency observation perspective, "inter-frame motion jump" is selected from the negative descriptor set corresponding to the temporal motion observation perspective, and "geometric structure distortion" is selected from the negative descriptor set corresponding to the physical rationality observation perspective. Based on the above selected descriptors, supervision labels for each observation agent are generated, and the authenticity judgment result label in the supervision label of the adjudication agent is set to "fake".

[0136] The process shown in S601-S602 above automatically selects and generates supervision labels based on the different authenticity types of sample video frames by using the pre-set set of positive and negative descriptive words corresponding to each observation perspective in the lexicon. This achieves automated and large-scale construction of supervision labels and ensures the uniformity and parsability of the output style under different forgery types.

[0137] like Figure 7 The diagram shown is a seventh flowchart of a deepfake video detection method based on multi-agent collaborative forensics provided in this application embodiment, including the following steps.

[0138] S701: Freeze the underlying network parameters of each observer agent and adjudicator agent.

[0139] The bottom-level network parameters are the parameters of the network layers closest to the input layer in a multimodal large model. These parameters are responsible for the general visual and text understanding capabilities of the multimodal large model. Freezing the bottom-level network parameters means keeping them unchanged during training and not updating gradients.

[0140] It's important to note that freezing the underlying network parameters preserves the general visual and text understanding capabilities of the multimodal large model during subsequent fine-tuning. Since both the observation and adjudication agents are built upon the multimodal large model, its underlying network parameters have already been pre-trained on large-scale data, demonstrating strong general visual and text understanding capabilities. Freezing the underlying network parameters during supervised fine-tuning avoids overfitting or forgetting pre-trained knowledge due to the limited scale of fine-tuning data.

[0141] S702: Based on the supervision labels, fine-tune the top-level network parameters of each observation agent and the adjudicator agent so that the difference between the visual evidence output by each observation agent and the corresponding supervision label meets the preset conditions, and the difference between the judgment basis and the authenticity judgment result output by the adjudicator agent and the corresponding supervision label meets the preset conditions.

[0142] The top-level network parameters are those of the network layers closest to the output layer in a large multimodal model. The preset conditions are criteria used to determine whether the difference between the model output and the supervision labels meets the training objective.

[0143] In some examples, the preset conditions can be that the cross-entropy loss between the model output and the supervision label is less than a preset loss threshold, or the accuracy between the model output and the supervision label is greater than a preset accuracy threshold, or the semantic similarity between the model output and the supervision label is greater than a preset similarity threshold.

[0144] The process described in S701-S702 freezes the underlying network parameters of each observer agent and the adjudicator agent, and only fine-tunes the top-level network parameters. This maintains the basic visual understanding ability of each agent while enabling each agent to efficiently learn specific role divisions and output formats, thus avoiding the problems of high training costs and model capability degradation caused by full parameter fine-tuning.

[0145] like Figure 8 The diagram shown is an architectural schematic of a deepfake video detection device based on multi-agent collaborative forensics provided in an embodiment of this application, including the following units.

[0146] The input acquisition unit 100 is used to acquire input video frames.

[0147] The observation and evidence collection unit 200 is used to observe and collect evidence from the input video frame from different observation perspectives by multiple pre-constructed observation agents, and generate visual evidence corresponding to the observation perspectives of each observation agent; wherein, each observation agent is isolated from each other during the observation and evidence collection process, and the evidence collection results are not visible to each other, and each observation agent does not output the authenticity judgment result.

[0148] Optionally, the multiple observation agents include a texture detail observation agent, an illumination consistency observation agent, a temporal motion observation agent, and a physical plausibility observation agent; wherein, the texture detail observation agent, the illumination consistency observation agent, the temporal motion observation agent, and the physical plausibility observation agent are constructed through corresponding role prompts, and each observation agent determines its corresponding observation perspective based on the corresponding role prompts.

[0149] Optionally, the observation and evidence collection unit 200 is specifically used for: observing and collecting evidence from the perspective of texture detail observation agent, generating texture detail visual evidence; texture detail visual evidence is used to characterize abnormal skin texture information and abnormal edge transition information in the input video frame; observing and collecting evidence from the perspective of illumination consistency observation agent, generating illumination consistency visual evidence; illumination consistency visual evidence is used to characterize abnormal light source direction information, abnormal shadow information, and abnormal highlight information in the input video frame; observing and collecting evidence from the perspective of temporal motion observation agent, generating temporal motion visual evidence; temporal motion visual evidence is used to characterize abnormal inter-frame motion continuity information and abnormal facial dynamic naturalness information in the input video frame; observing and collecting evidence from the perspective of physical rationality observation agent, generating physical rationality visual evidence; physical rationality visual evidence is used to characterize abnormal geometric structure information and abnormal physical law information in the input video frame.

[0150] The summary generation unit 300 is used to generate a summary observation report based on various visual evidences.

[0151] Optionally, the summary generation unit 300 is specifically used for: acquiring visual evidence output by each observation agent; the visual evidence is structured text data; the structured text data includes evidence type identifier, evidence description information and evidence location information; splicing the visual evidence according to a preset splicing order to generate a summary observation report; wherein the summary observation report is plain text data.

[0152] The adjudication reasoning unit 400 is used to input the observation report summary into the pre-built adjudication agent, and the adjudication agent performs reasoning on the observation report summary to obtain the basis for judgment; wherein, the adjudication agent does not acquire the input video frames.

[0153] Optionally, the adjudication reasoning unit 400 is specifically used to: input the observation report summary into a pre-built adjudication agent, perform text semantic understanding and logical reasoning on the observation report summary through the adjudication agent, identify the correlation and contradiction relationships between various visual evidences, and generate judgment criteria based on the correlation and contradiction relationships.

[0154] Result generation unit 500 is used to generate a true or false judgment result based on the judgment criteria.

[0155] The agent training unit 600 is used for: acquiring sample video frames and their authenticity types; constructing supervisory labels based on the authenticity types of the sample video frames and a pre-set lexicon; performing supervised fine-tuning on each observation agent and adjudicator agent based on the supervisory labels, so that each observation agent stably outputs corresponding visual evidence, and the adjudicator agent stably outputs judgment criteria and authenticity judgment results; and performing reinforcement learning fine-tuning on each observation agent and adjudicator agent through a group relative policy optimization algorithm. The reward function for reinforcement learning fine-tuning includes a length reward and a format reward. The length reward constrains the length of the visual evidence output by each observation agent and the length of the judgment criteria output by the adjudicator agent. The format reward constrains each observation agent and adjudicator agent to output according to a preset output format.

[0156] Optionally, the agent training unit 600 is specifically used to: select target descriptive words from a pre-set lexicon based on the authenticity type of the sample video frames; the authenticity type includes real type, face-swapping type, face replay type, and fully synthetic face type; the pre-set lexicon includes a set of positive descriptive words and a set of negative descriptive words; the positive descriptive words in the positive descriptive word set are used to characterize the natural and coordinated visual appearance of real faces under various viewing angles; the negative descriptive words in the negative descriptive word set are used to characterize the abnormal traces that appear on fake faces under various viewing angles; and generate supervision labels corresponding to the viewing angles of each observing agent based on the target descriptive words.

[0157] Optionally, the agent training unit 600 is specifically used to: freeze the underlying network parameters of each observation agent and adjudicator agent; and fine-tune the top-level network parameters of each observation agent and adjudicator agent based on the supervision label, so that the difference between the visual evidence output by each observation agent and the corresponding supervision label meets the preset conditions, and the difference between the judgment basis and the authenticity judgment result output by the adjudicator agent and the corresponding supervision label meets the preset conditions.

[0158] In summary, this scheme employs multiple observation agents to blindly observe and collect evidence from input video frames from different perspectives. An adjudicating agent then independently infers from the observation reports compiled from plain text to obtain the basis for judgment and the result of authenticity determination. This approach restructures the detection process into a separate architecture of independent evidence collection by multiple agents and plain text adjudication, making the decision-making chain transparent and traceable. It solves the technical problem of existing technologies that directly output probability scores in an end-to-end black-box manner, failing to provide auditable judgment criteria. By conducting parallel blind observations by four observation agents—texture detail, lighting consistency, temporal motion, and physical plausibility—the comprehensiveness and independence of the evidence are ensured. The separation of observation and judgment functions, along with the fact that the adjudicating agent does not touch the original pixels, eliminates visual bias and guarantees the objectivity of evidence collection and adjudication. The adjudicating agent identifies the relationships and contradictions between evidence and generates judgment criteria, ensuring that the final authenticity determination result is accompanied by a traceable chain of reasoning. Through automated construction of supervised labels using a pre-built lexicon and a two-stage progressive training process using SFT+GRPO, the stability of the output format and the accuracy of counterfeit detection are balanced, meeting the requirements of identity authentication and content review scenarios for verifiable and attributable results.

[0159] This application also provides a computer-readable storage medium including a stored program, wherein the program executes the deepfake video detection method based on multi-agent collaborative forensics provided in this application.

[0160] This application also provides an electronic device, including a processor, a memory, and a bus. The processor and the memory are connected via the bus. The memory is used to store a program, and the processor is used to run the program. During program execution, the deepfake video detection method based on multi-agent collaborative forensics provided in this application is executed.

[0161] While several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0162] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.< / observation> < / observation> < / observation> < / observation> < / answer> < / answer> < / explanation>

Claims

1. A deepfake video detection method based on multi-agent collaborative forensics, characterized in that, include: Obtain the input video frame; Multiple pre-constructed observation agents are used to observe and collect evidence from the input video frames from different perspectives, generating visual evidence corresponding to the observation perspectives of each observation agent. During the observation and evidence collection process, each observation agent is isolated from the others and cannot see the evidence collection results of the others, and each observation agent does not output a true or false judgment result. Based on each of the aforementioned visual evidences, a summary observation report is generated; The observation report is summarized and input into a pre-built adjudicating agent, which then performs reasoning on the summary of observation reports to obtain a basis for judgment; wherein, the adjudicating agent does not acquire the input video frames; Based on the aforementioned criteria, a true / false determination result is generated.

2. The method according to claim 1, characterized in that, The plurality of observation agents include a texture detail observation agent, an illumination consistency observation agent, a temporal motion observation agent, and a physical plausibility observation agent; wherein, the texture detail observation agent, the illumination consistency observation agent, the temporal motion observation agent, and the physical plausibility observation agent are each constructed through corresponding role prompt words, and each of the observation agents determines its corresponding observation perspective based on the corresponding role prompt words; By employing multiple pre-constructed observation agents, the input video frames are observed and examined from different perspectives to generate visual evidence corresponding to the observation perspectives of each agent, including: The texture detail observation agent observes and collects evidence from the perspective of texture detail observation of the input video frame, generating texture detail visual evidence; the texture detail visual evidence is used to characterize skin texture abnormality information and edge transition abnormality information in the input video frame; The illumination consistency observation agent observes and collects evidence from the perspective of illumination consistency observation of the input video frame, generating illumination consistency visual evidence; the illumination consistency visual evidence is used to characterize light source direction anomaly information, shadow anomaly information, and highlight anomaly information in the input video frame; The temporal motion observation agent observes and collects evidence from the perspective of temporal motion observation of the input video frame, generating temporal motion visual evidence; the temporal motion visual evidence is used to characterize the abnormal information of inter-frame motion coherence and abnormal information of facial dynamic naturalness in the input video frame. The physical rationality observation agent observes and collects evidence from the perspective of physical rationality observation of the input video frame, generating physical rationality visual evidence; the physical rationality visual evidence is used to characterize the geometric structure anomaly information and physical law anomaly information in the input video frame.

3. The method according to claim 1, characterized in that, Based on the aforementioned visual evidence, a summary observation report is generated, including: The visual evidence output by each of the observation agents is acquired; the visual evidence is structured text data; the structured text data includes evidence type identifier, evidence description information, and evidence location information. According to the preset splicing order, the various visual evidences are spliced ​​together to generate the observation report summary; wherein, the observation report summary is plain text data.

4. The method according to claim 1, characterized in that, The observation reports are summarized and input into a pre-built adjudicating agent. The adjudicating agent then performs reasoning on the summarized observation reports to obtain judgment criteria, including: The observation reports are compiled and input into a pre-built adjudication agent. The adjudication agent performs text semantic understanding and logical reasoning on the observation reports to identify the correlations and contradictions between the various visual pieces of evidence. Based on the aforementioned relationships and contradictions, a judgment criterion is generated.

5. The method according to claim 1, characterized in that, The training process for the observation agent and the adjudication agent includes: Obtain sample video frames and their authenticity type; Based on the authenticity type of the sample video frames, and in conjunction with a pre-set vocabulary, supervisory labels are constructed. Based on the supervision label, each of the observation agents and the adjudication agents are fine-tuned to ensure that each of the observation agents stably outputs the corresponding visual evidence, and the adjudication agent stably outputs the judgment basis and the authenticity judgment result. The observation agent and the adjudicator agent are fine-tuned through reinforcement learning using a group relative policy optimization algorithm. The reward function of the reinforcement learning fine-tuning includes a length reward and a format reward. The length reward is used to constrain the length of the visual evidence output by each observation agent and the length of the judgment basis output by the adjudicator agent. The format reward is used to constrain each observation agent and the adjudicator agent to output according to a preset output format.

6. The method according to claim 5, characterized in that, Based on the authenticity type of the sample video frames, and in conjunction with a pre-built vocabulary, supervisory labels are constructed, including: Based on the authenticity type of the sample video frames, target descriptive words are selected from a pre-set lexicon. The authenticity type includes real type, face-swapping type, face replay type, and fully synthetic face type. The pre-set lexicon includes a positive descriptive word set and a negative descriptive word set. The positive descriptive words in the positive descriptive word set are used to describe the natural and coordinated visual appearance of a real face under various viewing angles. The negative descriptive words in the negative descriptive word set are used to describe the abnormal traces that appear on a fake face under various viewing angles. Based on the target descriptor, supervision labels corresponding to the observation perspectives of each of the observation agents are generated.

7. The method according to claim 5, characterized in that, Based on the supervision labels, supervised fine-tuning is performed on each of the observing agents and the adjudicating agents, including: Freeze the underlying network parameters of each of the observation agents and the adjudicating agents; Based on the supervision labels, the top-level network parameters of each observation agent and the adjudication agent are fine-tuned so that the difference between the visual evidence output by each observation agent and the corresponding supervision label meets the preset conditions, and the difference between the judgment criteria and the authenticity judgment result output by the adjudication agent and the corresponding supervision label meets the preset conditions.

8. A deepfake video detection device based on multi-agent collaborative forensics, characterized in that, include: The input acquisition unit is used to acquire input video frames; The observation and evidence collection unit is used to observe and collect evidence from the input video frame from different observation perspectives by multiple pre-constructed observation agents, and generate visual evidence corresponding to the observation perspectives of each observation agent; wherein, each observation agent is isolated from each other during the observation and evidence collection process, and their evidence collection results are not visible to each other, and each observation agent does not output a true or false judgment result. The summary generation unit is used to generate a summary observation report based on each of the aforementioned visual evidences; The adjudication reasoning unit is used to input the observation report summary into a pre-built adjudication agent, and the adjudication agent performs reasoning on the observation report summary to obtain the judgment basis; wherein, the adjudication agent does not acquire the input video frame; The result generation unit is used to generate a truth or falsehood determination result based on the judgment criteria.

9. A storage medium, characterized in that, The storage medium includes a stored program, wherein the program is executed by a processor to perform the deepfake video detection method based on multi-agent collaborative forensics as described in any one of claims 1-7.

10. An electronic device, characterized in that, include: Processor, memory, and bus; The processor and the memory are connected via the bus; The memory is used to store the program, and the processor is used to run the program, wherein the program is executed by the processor to perform the deepfake video detection method based on multi-agent collaborative forensics as described in any one of claims 1-7.

Images