Deeply-forged face video frame-level positioning method and system based on weak supervised learning

Abstract

The invention discloses a deeply-forged face video frame-level positioning method and system based on weak supervised learning, and the method comprises the steps: firstly constructing a training set with a video as a unit, carrying out the data enhancement of a frame-level sample, and generating an enhanced view pair; secondly, splicing the enhanced view pair and inputting the spliced enhanced view pair into a depth forgery detection model to obtain and generate fusion enhanced frame-level features; then, intra-class contrast learning loss, time sequence consistency constraint loss and frame weight loss are constructed, and a deep forgery detection model is trained based on fusion-enhanced frame-level feature joint optimization. And finally, inputting a video to be detected into the deep counterfeiting detection model to output the frame-level confidence, judging whether the video is a forged video or not, and realizing frame-level counterfeiting positioning. According to the method, video-level detection and frame-level positioning are effectively realized, meanwhile, the influence of label noise in part of forged videos is relieved, and the generalization and robustness of the system are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method and system for frame-level localization of deepfake face videos based on weakly supervised learning. Background Technology

[0002] With the rapid development of generative artificial intelligence (AIGC), various AI-generated content has been widely disseminated in social networks, film and television production, and entertainment applications, demonstrating significant application value in these fields. Deepfakes, as a typical subtype of AIGC, are frequently used for telecommunications fraud, dissemination of false information, defamation, and interference with judicial evidence assessment, posing a serious threat to public safety, personal privacy, and the credibility of digital media. As technology continues to advance, attackers can even tamper with faces in only a few key frames of a video, thereby increasing the concealment of the forgery and significantly increasing the difficulty of detection. Such partial tampering often introduces label noise during model training, and the video may undergo post-processing operations such as compression, cropping, and scaling during dissemination, further reducing detection accuracy. Meanwhile, obtaining accurate frame-level manual annotations is not only costly but also difficult to cover diverse forgery types, limiting the adaptability of existing supervised detection methods in real-world, complex scenarios.

[0003] Existing deepfake detection methods can be mainly divided into three categories: supervised, unsupervised, and weakly supervised. Supervised methods rely on a large amount of high-quality frame-level labeled data and can achieve high detection accuracy on known forgery types. However, they perform poorly on some tampered videos and novel forgery types, and frame-level labeling is extremely costly, limiting the practical application of these methods. Unsupervised methods distinguish between real and fake videos through frequency domain features, pixel consistency, or artifact analysis, reducing the dependence on labeling. However, they struggle to capture deep semantic differences and lack robustness to complex forgeries or videos that have undergone compression, cropping, scaling, or other post-processing. Existing weakly supervised deepfake detection methods typically use video-level labels for training, but they mostly remain at the overall video level, failing to fully explore the semantic differences between frames and making it difficult to achieve an effective transition from video-level supervision to frame-level detection and localization. In some tampered videos, real and forged frames coexist. Directly using video-level labels can lead to a shift in the supervision signal, weakening the model's ability to identify key forged frames. Furthermore, post-processing operations such as compression and cropping further affect the distribution of features between frames, reducing detection robustness and accuracy.

[0004] Therefore, existing methods generally have the following limitations: First, they lack a frame-level learning mechanism for video-level labels, making it impossible to achieve frame-level detection and forgery localization under weak supervision; second, they do not fully utilize the semantic differences between frames, making it difficult to accurately identify the forgery features of some tampered frames; third, they are easily affected by interference from real frames when training directly using video-level labels, resulting in supervision noise and feature confusion; and fourth, under complex post-processing conditions, the model's generalization and robustness are insufficient, limiting its practical application effectiveness. Summary of the Invention

[0005] This invention addresses the challenge of achieving accurate frame-level detection and localization in existing deepfake detection methods under video-level labeling conditions. It proposes a frame-level localization method and system for deepfake face videos based on weakly supervised learning. This method constructs a detection framework combining frame-level dynamic weighting and intra-class contrastive learning, achieving frame-level detection and localization with only video-level labels. Through a multi-instance structure design that uses the video as the overall supervision unit and the frame as the basic analysis unit, it achieves a mapping from video-level labels to frame-level detection and localization. Furthermore, by combining intra-class contrastive learning and dynamic weighting mechanisms, it effectively models inter-frame heterogeneity, suppresses supervision bias, and significantly improves the model's robustness and detection accuracy in complex scenarios. During training, the input video frame samples are first augmented using various methods, including noise addition, compression, rotation, and scaling, to generate semantically consistent augmented view pairs, improving the model's adaptability to post-processing forgery and diverse distortions. Next, frame-level feature extraction is performed on the augmented views, and temporal feature fusion is conducted based on the residuals of adjacent frames. This explicitly models inter-frame motion discontinuities and forged abrupt changes, generating fused and augmented frame-level features through residual-aware gating and attention modules. Subsequently, intra-class contrastive learning constraints are used to aggregate features of similar frame samples in the feature space, constructing a stable and compact real sample feature space, providing reliable semantic anchors for the identification and localization of forged frames. Simultaneously, a temporal consistency constraint loss is introduced, constraining the smoothness between adjacent frame features to ensure the model maintains a stable response to natural inter-frame changes in real videos while remaining sensitive to forged abrupt changes, further improving the accuracy and robustness of frame-level detection. Finally, a frame-level dynamic weighting mechanism is introduced, adaptively adjusting weights based on the similarity between frame features and real clustering prototypes, mining frame-level forgery clues from video-level labels to achieve accurate detection and localization of partially tampered frames. This invention achieves an effective transformation from video-level supervision to frame-level detection under weak supervision conditions lacking precise frame-level annotations, significantly improving the robustness and generalization ability of deepfake detection models, effectively balancing detection performance and annotation costs, and meeting the high-efficiency detection needs in complex real-world application scenarios.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0007] A frame-level localization method for deep fake face videos based on weakly supervised learning includes the following steps:

[0008] Step 1: Construct the training set samples. The training set is constructed on a video-by-video basis. Original videos containing faces (including real videos and fake videos) are cropped into frame-level samples. Each video corresponds to a "packet," and each frame sample within the packet corresponds to an "example." Specifically, all frames within the real video packet are real frames and are uniformly labeled with real labels. The frame samples within the fake video packet contain both tampered fake frames and authentic real frames. However, since only video-level labels are used for training, all frames within this packet need to be uniformly labeled with fake labels, thus introducing label noise into the training set.

[0009] Building upon this, data augmentation is performed on frame-level samples to construct multi-view sample pairs for contrastive learning. Specifically, two different data augmentation methods are randomly applied to each frame-level example to generate view pairs that differ in appearance but are semantically consistent. This augmentation process improves the model's discriminativeness and robustness in the feature space, providing diverse inputs and feature consistency constraints for subsequent residual feature augmentation and temporal modeling.

[0010] Step 2: The enhanced view generated in Step 1 is concatenated and input into the backbone neural network of the deep forgery detection model for frame-level feature extraction. Then, the residual information is obtained by subtracting the features of adjacent frames to explicitly model the motion discontinuities and forged abrupt changes between frames. Next, the residual information and frame-level features are concatenated along the channel dimension and input into the residual-aware gating and attention fusion module to generate fused and enhanced frame-level features. This process preserves the spatial appearance features of the frames while enhancing the model's ability to detect and distinguish forged abrupt changes.

[0011] Step 3: Use the fused and enhanced features obtained in Step 2 to train the deep forgery detection model. For the extracted fused and enhanced frame-level features, an intra-class contrastive learning loss is used to bring the features of frames of the same class (real frames - real frames) closer together, while frames of different classes (real frames - forged frames) or uncertain classes are not forced to be brought closer together, thereby forming a stable and compact feature space of real samples.

[0012] Building upon this, a temporal consistency constraint loss is introduced. By calculating the smoothness loss between features of adjacent frames, the model's over-response to inter-frame abrupt changes in real videos is suppressed, and its sensitivity to abruptly changed frames of forged segments is enhanced. Finally, a frame weight loss is designed. Frame weights are dynamically allocated based on the cosine similarity between the frame corresponding to the forged label and the prototype features obtained from clustering real samples. The weights are adjusted with the similarity to amplify the contribution of key forged frames in the loss calculation.

[0013] The intra-class contrastive learning loss, temporal consistency constraint loss, and frame weight loss are jointly optimized. The backbone network is trained through backpropagation, which enables the model to effectively suppress noise frame interference under weak supervision, while improving the ability to detect and locate some forged frames.

[0014] Step 4: Frame-level Detection Result Output and Forgery Localization. After training, the video to be detected is input into the deep forgery detection model, which outputs frame-level confidence scores. The backbone network extracts feature representations for each frame of the video, and the classification head outputs frame-level confidence scores based on the fused and enhanced frame-level features. If any frame has a confidence score greater than a threshold, the video is determined to be forged. Based on this, the frame-level confidence scores are arranged sequentially according to the video frames, and the frame sequence intervals with continuously higher confidence scores than the threshold are identified. These frame sequence intervals correspond to suspected forged segments and are represented by their start and end frame numbers, thus achieving frame-level forgery localization. Finally, the system returns the detection results to the user, including a video authenticity label, the forged frame sequence interval, and the frame-level confidence score.

[0015] Preferably, the backbone neural network adopts the VisionTransformer structure, and contrastive learning loss function, temporal consistency loss and weight loss function are introduced as constraints during the training process.

[0016] Preferably, the real sample clustering employs a Gaussian mixture model based on cosine similarity to suppress feature shifts in video frames caused by low-level factors such as lighting variations and skin color differences. The cluster centers generated by clustering serve as prototypes of real samples to construct stable reference feature representations, enabling dynamic evaluation of the similarity between forged and real frames during training. This achieves frame-level weight allocation and optimization of forgery detection under weak supervision.

[0017] Preferably, the aggregation method of the video-level forgery confidence score is to take the maximum value among the frame-level confidence scores, that is, the frame with the highest confidence score in the video is taken as the video-level forgery confidence score, so as to reflect the principle of "if there is even one frame of forgery, the video is determined to be forged", thereby achieving sensitive detection of some forged videos.

[0018] On the other hand, the present invention provides a deep fake face video frame-level localization system based on weakly supervised learning, comprising the following modules:

[0019] The training set construction module is used to build the training set on a video-by-video basis, splitting the video into frame-level samples and organizing them into a "bundle-example" data structure required for multi-instance learning. For each frame sample, data augmentation operations are performed, including noise perturbation, image compression, rotation, scaling, etc., generating multi-view augmented frame sample pairs. This module provides consistency constraints between the base samples and features, providing input for subsequent frame feature extraction and weakly supervised training.

[0020] The feature extraction and fusion module is used to input the enhanced view generated in step 1 into the depth forgery detection model and extract frame-level features through the backbone network. Then, attention fusion and gating operations are performed on the residual information obtained by subtracting the frame-level features from the features of adjacent frames to generate fused and enhanced frame-level features.

[0021] The joint optimization training module is used to construct multiple loss functions based on the fused and enhanced frame-level features and perform joint optimization training. This module calculates the intra-class contrastive learning loss to bring features of frames within the same class closer together; it calculates the temporal consistency constraint loss, weighting the differences in features between consecutive frames using a sliding window and residual information; and it calculates the frame weight loss, dynamically assigning frame weights based on the similarity between the cluster centers of forged frames and real samples. These three types of losses form a weighted total loss function for end-to-end training of the deepfake detection model.

[0022] The forgery detection and localization module is used to perform inference on the input video after training and output the forgery confidence score for each frame. It determines whether the entire video is forged based on the frame-level confidence score and locates forged regions at the frame level, supporting structured display of the forged frame distribution.

[0023] This invention has the following characteristics and beneficial effects:

[0024] (1) A deep forgery detection and localization learning framework based on weak supervision is proposed. It can realize frame-level authenticity detection and forgery segment localization by relying only on video-level labels, which breaks through the limitation of traditional methods that can only make video-level judgments and significantly improves the detection capability of some forged videos.

[0025] (2) Introduce a residual perception mechanism to construct temporal residual information through the feature difference between adjacent frames, guide the model to focus on the inter-frame mutation region, highlight the key forgery region, and improve the model's response sensitivity to forged frames.

[0026] (3) Based on the residual-driven k-frame sliding window smoothing constraint, the weighted temporal consistency loss is calculated for the fusion features of consecutive k frames. The gradient response of the fake mutation frame is amplified by the residual-aware dynamic weighting function, while suppressing the influence of natural changes between real frames. This enables the model to learn the temporal features and mutation patterns of the fake video, thereby further improving the positioning accuracy of the key frame.

[0027] (4) Combining the dynamic frame weight allocation strategy of real sample prototypes, the frame contribution is adaptively adjusted according to the feature similarity between the frame and the prototype, which effectively alleviates the noise labeling problem of some fake videos under weak supervision and improves detection robustness and generalization ability.

[0028] (5) The system design covers the entire process of “preprocessing, feature extraction, weight allocation, aggregation and localization”, supports dynamic iteration of modules, can adapt to new forgery methods, and achieves high robustness and wide applicability. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart illustrating the weakly supervised detection and localization model in an embodiment of the present invention.

[0031] Figure 2 This is a schematic diagram of the video detection process in an embodiment of the present invention. Detailed Implementation

[0032] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0033] This invention proposes a multi-instance learning weakly supervised framework that treats videos as "packets" and frames as "instances," relying solely on video-level labels for training to achieve authenticity detection and keyframe localization for each frame in a video. During this process, by dynamically allocating frame weights based on real sample prototypes, the problem of label noise present in some fake videos is effectively mitigated, thereby improving the model's robustness and generalization ability in frame-level detection and localization tasks.

[0034] A feature extraction mechanism of "data augmentation and contrastive learning" is constructed. By simulating the augmentation method and contrastive loss of post-processing, the model learns robust frame features and improves the model's resistance to post-processing interference.

[0035] By incorporating real sample clustering and weighted loss into the training, adjusting frame weights with cluster prototypes as a reference, and then aggregating video-level confidence through weighted averaging, the system is adapted to real detection scenarios, improving overall robustness and practicality.

[0036] In one aspect, this invention provides a frame-level localization method for deepfake face videos based on weakly supervised learning, comprising the following steps:

[0037] Step 1: Construct a training set based on videos, and perform data augmentation on frame-level samples to generate augmented view pairs.

[0038] Step 2: Concatenate the enhanced view with the input depth forgery detection model and obtain the generated fused and enhanced frame-level features.

[0039] Step 3: Based on the fused and enhanced frame-level features, construct intra-class contrastive learning loss, temporal consistency constraint loss, and frame weight loss, and jointly optimize and train the deep forgery detection model.

[0040] Step 4: After training is complete, input the video to be detected into the deep fake detection model to output frame-level confidence, determine whether the video is fake, and achieve frame-level fake location.

[0041] Preferably, in step 1, the original video dataset is obtained. ,in This refers to the original video, in Chinese. These represent real and counterfeit labels, respectively. This indicates the number of videos; face cropping is performed on the videos using Dlib to obtain a frame-level image dataset. ,in This represents the cropped video frame. Indicates the number of video frames. Tags are inherited from video-level tags, but may contain tag noise in fake videos.

[0042] For frame-level samples Data augmentation ,in This includes methods such as noise enhancement, compression, flipping, rotation, and scaling; two enhancement methods are randomly selected for each sample to generate view pairs. This is to construct the final frame-level training set.

[0043] Preferably, in step 2, the view pairs are stitched together and input into the backbone neural network. Extracting high-dimensional feature representations And mapped to a low-dimensional feature representation through a projection head. That is, frame-level feature representation.

[0044] Furthermore, in order to capture the dynamic change features between video frames, adjacent frames are... , Features , Calculate the time-series residual information:

[0045]

[0046] Residual information View features of the corresponding frame The final fused features are generated through residual-aware gating and attention mechanisms:

[0047]

[0048]

[0049] in, Represents the absolute value of the residual, used to reflect the intensity of the mutation; This is the concatenated vector of the residual and frame-level features. and These are the weight matrix and bias of the gated layer, respectively; The sigmoid function is used to limit the gating coefficient to a certain value. interval; This represents a residual-guided attention fusion operation that uses residual information to adjust the time-sensitive response of view features; the final fused feature is enhanced. The weighted combination of residual information and original view features is controlled by gating coefficients, thereby achieving dynamic perception and enhancement of keyframe forgery mutations.

[0050] Preferably, in step 3, all fusion enhancement features are processed. Constructing a similarity matrix involves: using training batches as units, assuming each batch contains... Each sample has a fused enhancement feature. ( (as feature dimension), forming the feature matrix Calculate the cosine similarity of all sample pairs within a batch, for the . Features and the Features The cosine similarity is Arrange the cosine similarity of all sample pairs to form Similarity matrix Then divide all elements in the similarity matrix by the temperature coefficient. The scaled similarity matrix is ​​obtained. Element is .

[0051] Simultaneously, a mask matrix is ​​constructed based on the sample labels. When sample With sample When samples belong to the same category, the corresponding element in the mask matrix is ​​set to 1; otherwise, it is set to 0. This method is used to filter samples of the same category based on their features. The model training is constrained by a contrastive learning loss function. The contrastive learning loss function is defined as follows:

[0052]

[0053] in, This represents the number of samples in the current training batch. It is a mask matrix; Represents each sample The corresponding number of samples of the same type; The elements are the scaled similarity matrix; the denominator is the sum of the exponential similarities of all other samples in the batch (excluding itself). These constraints enable the aggregation of similar features and the separation of dissimilar features, thus obtaining a robust frame-level feature representation.

[0054] Preferably, in step 3, the frame-level examples in the training set are processed. First, the video frame features corresponding to the real labels are clustered using a Gaussian mixture model based on cosine similarity to suppress feature shifts caused by low-level factors such as lighting changes and skin color differences. The cluster centers are then selected as prototypes of the real samples. :

[0055]

[0056] in Represents the set of real labels for all real samples. Used as a reference feature for real samples.

[0057] Subsequently, the cosine similarity between the frame features corresponding to the fake label and the real sample prototype is calculated as a weight:

[0058]

[0059] in, The frame weight is fixed for the frame corresponding to the real label. The frame weight corresponding to the fake label is determined by the cosine similarity, which is used to reflect the contribution of the frame to the video-level label.

[0060] In calculating weight-based loss At that time, first, based on frame-level feature weights and Weighted similarity is calculated for samples to the fused and enhanced frame-level features. Based on this, a weighted similarity matrix is ​​constructed, and then the weighted loss is calculated using this matrix.

[0061]

[0062] in, As the focus term, it contributes more to the difficult samples where the model currently has "low confidence".

[0063] Furthermore, to suppress the model's over-response to natural inter-frame differences in real videos, while amplifying the abrupt changes in forged segments, for continuous... Frame characteristics and adjacent frame residual information Calculate the timing consistency loss:

[0064]

[0065] in, The number of video frames. It is the L2 norm. This is a residual-aware dynamic weighting function used to amplify the gradient response of abruptly changed frames.

[0066]

[0067] in, residual information of adjacent frames Norm, measures the magnitude of abrupt changes; This is the maximum value of the residuals of all adjacent frames in the current video sequence, used for normalization to ensure that the weighted values ​​are within a reasonable range; This is a hyperparameter that controls the amplification of the weighting coefficients by the residual magnitude. When the residual of a frame is large (potentially a forged abrupt change frame), its normalized value is high. A weighting factor greater than 1 makes the frame contribute more to the temporal smoothing loss or gradient; when the residual is small (stationary frame or true frame), the weighting factor is close to 1 to avoid weakening the original gradient, thus maintaining the smoothness of the true frame.

[0068] During the training phase, the total loss function is formed by weighted combination of intra-class contrastive loss, weighted loss, and temporal consistency loss:

[0069]

[0070] in For intra-class contrast loss, For weight loss, For time-series consistency loss, , and This is a hyperparameter used to adjust the relative importance of the three.

[0071] Preferably, in step 4, the video to be detected is acquired. , its frame set Input the trained detection model Extract frame-level features and calculate the forgery confidence score for each frame. ,in Subsequently, video-level determination is performed based on frame-level confidence: when any frame in the video... Above the threshold If the video is fake, it is determined to be genuine; otherwise, it is determined to be authentic and a video-level tag is obtained. :

[0072]

[0073] in This indicates the video is fake. This indicates that the video is authentic. This strategy ensures that even if only some frames of the video are tampered with, the detection model can still identify it as a fake video, thereby improving the sensitivity and detection accuracy of partially fake videos.

[0074] Based on this, frame-level confidence Arranged by time series, it is possible to identify those with confidence levels continuously above a threshold. The frame sequence intervals, which correspond to suspected forged segments, are represented by their start and end frame numbers to achieve frame-level forgery location.

[0075] Example:

[0076] This embodiment provides a deep fake face video frame-level localization method based on weakly supervised learning, which consists of two parts: model training stage and model detection stage.

[0077] (1) Model Training Phase: The main purpose of the model training phase is to build a detection model adapted to weakly supervised scenarios. By processing noise labels in some fake videos and simulating post-processing interference in video propagation, the model learns robust frame-level features and video-level decision-making logic, ultimately achieving accurate identification of deepfake videos in complex scenarios. At the same time, it avoids excessive reliance on high-quality frame-level annotations, balancing annotation costs and detection performance, such as... Figure 1 As shown, the specific training steps are as follows:

[0078] (1.1) Training set sample construction: The purpose of training set sample construction is to transform the original video data into standardized data that conforms to the "package-example" framework of multi-instance learning, to provide basic materials for model training, and to clarify the relationship between frame-level labels and video-level labels and potential noise problems.

[0079] (1.1.1) First, obtain the original video dataset and determine the labeling rules to construct the original video dataset. in Represents a single video sample. Indicates the number of videos. These represent the labels for real and fake videos, respectively. These labels will serve as the initial labeling basis for subsequent frame-level samples. It should be noted that some key frames in fake videos may be tampered with, resulting in noise in subsequent frame-level labels.

[0080] (1.1.2) Then, frame extraction and face cropping are performed on the original video to construct a frame-level dataset. The Dlib tool is then used to process the original video. The process begins by cropping frame-level images containing faces from the video, resulting in a frame-level image dataset. ,in For a single frame-level image sample, Indicates the number of video frames. Tags are inherited from video level tags However, due to partial alteration of the forged video, the corresponding frame-level tags of the forged video were... It may not be consistent with the authenticity of the actual frame, that is, there is label noise.

[0081] (1.1.3) Finally, the "package-example" structure is defined, the multi-example learning data format is clarified, and each original video is... All corresponding frame-level images Grouped into a "bag", a single frame-level image As an "instance", each "package" corresponds to a unique video-level tag. This forms the training data structure required for multi-instance learning, laying the foundation for subsequent weakly supervised training of models based on video-level labels.

[0082] (1.2) Frame-level data augmentation and feature extraction: In order to enhance the model's adaptability to video post-processing interference, for each frame Apply enhancement functions This includes operations such as noise addition, compression, rotation, and scaling, and randomly selects two enhancement methods to generate paired views. This is used to simulate various interferences in video propagation. The enhanced view is then fed into the backbone neural network. First extract high-dimensional features Then, it is mapped into low-dimensional features through the projection head. This provides a foundation for subsequent frame-level feature learning.

[0083] (1.3) To simultaneously capture spatial appearance features and temporal variation features, an inter-frame residual-guided fusion mechanism is introduced. This mechanism is used to fuse features from adjacent frames. Calculate residual information This is used to highlight abrupt changes in keyframes. Residual information and frame-level features are treated as two branches, and a final enhanced frame-level feature is generated through residual-aware gating and attention fusion. This enhances the perception of forged mutations and temporal discontinuities. An intra-class contrastive loss is constructed based on this enhanced frame-level feature fusion. This allows for the aggregation of features from frames of the same category, forming a stable and compact feature space for real samples, providing a reliable reference for identifying forged frames.

[0084] (1.4) Real Sample Clustering and Dynamic Frame Weight Allocation. Single-class clustering is performed on the features of real frames to obtain real sample prototypes. A Gaussian mixture model based on cosine similarity is used to suppress feature shifts caused by low-level factors such as lighting and skin color. Subsequently, the cosine similarity between the features of the forged frame and the prototype is calculated to obtain adaptive frame weights. This reflects the contribution of each frame to the video-level tags, and yields... The weight of the real frame is fixed at 1. This stage serves as a bridge between the feature space and the frame-level weights, providing a basis for temporal constraints.

[0085] (1.5) Temporal Consistency Constraint. To amplify the features of fake mutations and suppress natural differences between real frames, a temporal consistency constraint is introduced. Continuous Frame fusion features Residual with adjacent frames Jointly calculate timing consistency loss This mechanism enables the model to capture fake mutation patterns in the time dimension while ensuring the smoothness of real frames, thereby improving the accuracy of keyframe localization.

[0086] (1.6) Finally, the intra-class contrast loss is... Frame weight loss With timing consistency loss A weighted average is used to form a total loss. Optimizing this total loss allows the model to learn robust frame-level features under weak supervision, while simultaneously ensuring effective forged frame identification and noise frame suppression, achieving high-precision frame-level detection and forged segment localization. The entire training process forms a logical closed loop: from video input to frame-level feature extraction, then to residual-aware fusion, weight allocation, and temporal constraints, ultimately achieving a comprehensive understanding and accurate detection of some forged videos through joint loss.

[0087] (2) Model Detection Stage: The main purpose of the model detection stage is to use the trained detection model to process the input video to be detected, extract features, calculate confidence, and aggregate the results to finally output accurate video-level forgery detection conclusions, which are then fed back to the user to judge the authenticity of the video, such as... Figure 2 As shown, the specific process is as follows:

[0088] (2.1) Video processing and confidence calculation: Video processing and confidence calculation involves converting the video to be detected into a format that the model can process, and obtaining frame-level and video-level forgery confidence through model inference, providing data support for the final judgment.

[0089] (2.1.1) First, acquire the video to be detected and extract frame-level samples. (Acquire the video to be detected) Following the same method as constructing the training set, frame-level images containing faces are extracted from the video to be detected to form a set of frames to be detected. (in (This refers to the number of frames extracted from the video to be detected), ensuring that the sample format input to the model is consistent with that used in the training phase.

[0090] (2.1.2) Then, the frames to be detected are input into the model to calculate the frame-level forgery confidence. Each frame in the set of frames to be detected... Input the trained detection model The forgery confidence level of the frame is calculated using the backbone neural network of the model and the classification head. ,in , The closer a frame is to 1, the higher the probability that the frame is fake; the closer a frame is to 0, the higher the probability that the frame is real.

[0091] (2.1.3) Video-level forgery detection. Here, a "one frame forgery equals video forgery" strategy is adopted. A frame-level detection threshold is set. (Usually taken as 0.5), if any frame exists If the keyframes are altered, the entire video is determined to be fake; otherwise, the video is determined to be real. This method can ensure that even if only keyframes are altered, the model can accurately identify video forgery.

[0092] (2.2) The output of the detection results aims to transform the video-level confidence into an intuitive detection conclusion, and clearly inform the user of the authenticity of the video to be detected, while providing frame-level positioning information to ensure the usability and clarity of the detection results.

[0093] (2.2.2) Frame-level forgery segment location. The frame-level confidence level... Arranged by time series, identify continuous or discrete values ​​above the threshold. frame sequence interval As suspected forged segments, they are recorded with start and end frame numbers. Finally, the following structured information is returned to the user:

[0094]

[0095] This includes video-level tags. The system includes frame-level forgery intervals and the confidence level of each frame, ensuring that users can simultaneously understand the overall video status and the specific location of the forged frames.

[0096] On the other hand, the present invention provides a deep fake face video frame-level localization system based on weakly supervised learning, comprising the following modules:

[0097] The training set construction module is used to build training sets on a video-by-video basis and perform data augmentation on frame-level samples in the videos to generate augmented view pairs.

[0098] The feature extraction and fusion module is used to input the enhanced view pairs into the deep forgery detection model, extract frame-level features through the backbone network, and fuse them with residual information to obtain fused and enhanced frame-level features.

[0099] The joint optimization training module constructs intra-class contrastive learning loss, temporal consistency constraint loss, and frame weight loss based on fused and enhanced frame-level features. The total loss function formed by the weighting of the three is used to train the deep forgery detection model end-to-end.

[0100] The forgery detection and localization module is used to detect input videos after the deep forgery detection model has been trained, output frame-level confidence scores, determine whether the video is a forgery based on the frame-level confidence scores, and locate the forgery region at the frame level.

[0101] To verify the effectiveness of the proposed method in achieving frame-level detection and localization under weakly supervised conditions using video-level label training, ForgeryNet and FaceForensics++ (FF++) were selected as evaluation datasets. ForgeryNet is currently the largest DeepFake dataset, supporting four tasks: temporal forgery localization, spatial forgery localization, image forgery classification, and video forgery classification. It contains 2.9 million images and 221,247 videos, covering 15 different forgery methods. Only a portion of the frames in the forged videos on this dataset are altered, closely matching the experimental scenario of this invention. All forged videos are sourced from four mainstream face datasets: CREMA-D, RAVDESS, VoxCeleb2, and AVSpeech. To strictly control experimental conditions and avoid potential domain bias, both training and testing on ForgeryNet used real videos and their corresponding forged videos. The dlib tool was used to extract face regions from video frames, and frames were divided into real and forged frames based on the frame-level annotations provided by the dataset to quantitatively explore the impact of the proportion of partially forged frames on the method's performance. During the training phase, only video-level labels were used, without relying on frame-level annotations. To simulate some fake video scenarios, a noisy dataset was designed for the experiment. This involved mixing a certain proportion of real frames into the fake samples (e.g., adding 2 real frames out of every 10 frames, corresponding to a 20% noise rate), while the real samples remained pure. The FF++ dataset, derived from 1,000 high-quality videos in YouTube-8M, is one of the most widely used datasets for DeepFake detection and evaluation tasks. The experiment directly used standard data partitioning and annotation for training and testing, and reserved some real videos as reliable samples to ensure that there was no overlap between real frames and the mixed-in fake frames during training.

[0102] In the experiments, the VisionTransformer network was used as the backbone neural network, and all input images were uniformly adjusted to a size of 299×299. The Adam optimizer was used with a learning rate of 1e-3 and a weight decay of 1e-4. The batch size was 128, and the temperature coefficient τ was set to 0.07. The area under the curve (AUC) was used as the performance evaluation metric. An NVIDIA 3090Ti GPU was used to ensure efficient computational performance. Through the above experimental setup, the effectiveness and superiority of the proposed method in achieving frame-level detection and localization under video-level label training can be verified. Simultaneously, the impact of the proportion of forged frames on the model's robustness and detection performance can be quantitatively analyzed. It is worth noting that all experiments used the same hyperparameters under different noise rates, which is consistent with the characteristic that the proportion of forged frames in tampered videos is unknown in real-world scenarios.

[0103] Table 1. Comparison of AUC (%) under different noise rates using video-level labels on the ForgeryNet dataset.

[0104]

[0105] Table 2. Comparison of AUC (%) of the FF++ dataset under different noise rates using video-level labels.

[0106]

[0107] As can be observed from Tables 1 and 2, at a noise ratio of 20%, the video detection algorithms EfficientNetB4 and Xception can still maintain a certain performance in the video-level DeepFake detection task. However, as the noise ratio increases to 50% or above, their detection ability decreases significantly, especially under high noise conditions, they can hardly distinguish between real and fake videos.

[0108] In contrast, the method of this invention maintains stable and excellent detection performance across various proportions of forged frames (20%–80%), even under 80% high noise conditions, maintaining a high AUC. Experimental results show that this invention, through weakly supervised video-level label training combined with multi-instance learning and dynamic frame weight allocation, can effectively learn the heterogeneity features of video frames from noisy training data, achieving accurate differentiation between real and forged frames. This design ensures that the model can reliably output video-level detection results even in some forged video scenarios, while also supporting frame-level localization to accurately identify and label forged segments, fully demonstrating the robustness and generalization ability of the method under weak supervision.

[0109] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A deep fake face video frame-level localization method based on weakly supervised learning, characterized in that, The method comprises the following steps: Step 1: Construct a training set in units of videos, and perform data enhancement on frame-level samples to generate enhanced view pairs; Step 2: input the enhanced view pairs into a deep fake detection model, and obtain fused and enhanced frame-level features; Step 3: based on the fused and enhanced frame-level features, construct intra-class contrast learning loss, temporal consistency constraint loss and frame weight loss, and jointly optimize the deep fake detection model; Step 4: after training, input the video to be detected into the deep fake detection model to output frame-level confidence, determine whether the video is a fake video, and realize frame-level fake positioning.

2. The deepfake face video frame-level localization method based on weakly supervised learning according to claim 1, wherein, The training set is constructed in units of videos, and the original video containing a face is cropped into frame-level samples. The original video contains real videos and fake videos, each video corresponds to a package, and each frame sample in the package corresponds to an example. The real video package contains all real frames, which are uniformly labeled as real labels; The frame samples in the fake video package contain both tampered fake frames and un-tampered real frames, and all frames in the package are labeled as fake labels.

3. The method of claim 2, wherein, The data enhancement of the frame-level samples is specifically implemented by randomly applying two different data enhancement methods to each frame-level example to generate view pairs with different appearances but consistent semantics.

4. The method of claim 3, wherein the method is a deepfake video frame-level localization method based on weakly supervised learning. The step 2 is specifically implemented as: extracting feature representation of a backbone neural network of the enhanced view pair splicing input depth forgery detection model , and obtaining frame-level features through projection head mapping , then obtaining residual information by differencing the frame-level features of adjacent frames , then inputting the residual information and the frame-level features after being spliced in the channel dimension into a gate and attention fusion module to generate fused enhanced frame-level features.

5. The method of claim 4, wherein, The gating and attention fusion module is specifically implemented as follows: the residual information Taking absolute value and frame-level feature Splicing, and then generating a gating vector through linear activation mapping; the frame-level feature is spliced, and a gating vector is generated through linear activation mapping; the residual information After being spliced with the frame-level feature, the gating vector is weighted through attention fusion, an attention feature is obtained, the gating vector is inverted and weighted with the frame-level feature, and finally, the attention feature is added element by element to obtain the final fused and enhanced frame-level feature.

6. The method of claim 5, wherein the method is a deepfake face video frame-level localization method based on weakly supervised learning. In step 3, the fused and enhanced frame-level features obtained in step 2 are used to train the deep fake detection model. For the extracted fused and enhanced frame-level features, intra-class contrast learning loss is used to pull the features of frames in the same class closer, and does not forcibly pull the features of frames in different classes or uncertain classes closer, forming a real sample feature space; On this basis, the temporal consistency constraint loss is introduced, the smoothness loss between adjacent frame features is calculated to suppress the excessive response to inter-frame mutations in real videos; the frame weight loss is designed, the cosine similarity between the prototype features obtained by clustering according to the frame corresponding to the fake label and the real sample is taken as the frame weight, and the intra-class contrast learning loss is weighted; The intra-class contrast learning loss, the temporal consistency constraint loss and the frame weight loss are weighted and jointly optimized, and the backbone network is trained through back propagation; The backbone neural network adopts a Vision Transformer structure.

7. The method of claim 6, wherein the method is a deepfake face video frame-level localization method based on weakly supervised learning. The intra-class contrast learning loss, the temporal consistency constraint loss and the frame weight loss are specifically implemented as follows: Intra-class contrastive learning loss: Based on training batches, assuming each batch contains... Each sample has a fused enhancement feature. , The feature matrix is ​​constructed using the feature dimension. ; Calculate the cosine similarity of all sample pairs within a batch, for the . Features and the Features The cosine similarity is Arrange the cosine similarity of all sample pairs to form Similarity matrix Then divide all elements in the similarity matrix by the temperature coefficient. The scaled similarity matrix is ​​obtained. Element is Simultaneously, a mask matrix is ​​constructed based on the sample labels. When sample With sample When they belong to the same category, the corresponding position element of the mask matrix The value is 1 if the mask is set to 1 otherwise 0, thus filtering samples of the same type based on the mask matrix. and similarity matrix Calculate the intra-class contrastive learning loss function; Frame weight loss: process the frame-level examples in the training set, first use a Gaussian mixture model based on cosine similarity to perform single-class clustering on the fused and enhanced frame-level features corresponding to the real label, and select the clustering center as the prototype of the real sample; Subsequently, the cosine similarity between the fused and enhanced frame-level features corresponding to the fake label and the real sample prototype is calculated as a weight. , Represents the frame weight for the frame corresponding to the true label. The frame weight corresponding to the forged label is fixed at 1, and is determined by the cosine similarity; based on the frame-level feature weights... and We perform weighted calculations, calculate the weighted cosine similarity of samples to the fused and enhanced frame-level features, construct a weighted similarity matrix based on this, perform weighted intra-class contrastive learning loss calculations, and obtain the frame weight loss. Temporal consistency constraint loss: extract the fused and enhanced features of k consecutive frames and the residual information of adjacent frames from the video, calculate the Euclidean distance, and weight the Euclidean distance with a dynamic weighting function; traverse all windows and combine the frame weight to calculate the average value to form the final temporal consistency loss.

8. The deepfake face video frame-level localization method based on weakly supervised learning according to claim 7, wherein, The step 4 is specifically implemented as follows: after the training is completed, the backbone network is used to extract the fusion-enhanced frame-level features of each frame of the video, the classification head outputs frame-level confidence according to the fusion-enhanced frame-level features, and the video-level fake confidence is calculated through aggregation of the frame-level confidence, so as to determine that the video is fake, otherwise, it is determined to be real; on this basis, the frame-level confidence scores are arranged in the order of video frames, the frame sequence interval with continuous confidence higher than the threshold is identified, the frame sequence interval corresponds to the fake segment, and the start and end frame numbers are used to represent, so as to realize frame-level fake positioning; finally, the detection result is returned to the user, including the video-level fake confidence, the video real-fake determination label and the fake frame sequence interval.

9. The deepfake face video frame-level localization method based on weakly supervised learning according to claim 8, wherein, In step 4, the video to be detected is acquired , and the frame set thereof is input into the trained detection model , frame-level features are extracted and fused, and the fake confidence of each frame is calculated ; Subsequently, video-level determination is performed based on frame-level confidence: when any frame in the video... Above the threshold If the video is fake, it is determined to be genuine; otherwise, it is determined to be authentic and a video-level tag is obtained. : Frame-level confidence Arranged by time series, it is possible to identify those with confidence levels continuously above a threshold. The frame sequence intervals, which correspond to suspected forged segments, are represented by their start and end frame numbers to achieve frame-level forgery location.

10. A deepfake face video frame-level localization system based on weakly supervised learning, for implementing the deepfake face video frame-level localization method of any one of claims 1 to 9, characterized in that, The method comprises the following modules: A training set construction module is configured to construct a training set in units of videos, and perform data enhancement on frame-level samples in the videos to generate enhanced view pairs; A feature extraction and fusion module is configured to input the enhanced view pairs into a deep fake detection model, extract frame-level features through a backbone network, and fuse the frame-level features with residual information to obtain fusion-enhanced frame-level features; A joint optimization training module is configured to construct intra-class contrast learning loss, time sequence consistency constraint loss and frame weight loss based on the fusion-enhanced frame-level features, and perform end-to-end training on the deep fake detection model using a total loss function formed by weighting the three losses; A fake detection and positioning module is configured to, after the training is completed, detect an input video by using the deep fake detection model, output frame-level confidence, determine whether the video is a fake video according to the frame-level confidence, and realize frame-level fake region positioning.

Images