Deepfake video frame-level positioning method and system based on weakly supervised learning

Abstract

The application discloses a deep fake face video frame-level positioning method and system based on weak supervision learning, which firstly constructs a training set in units of videos, and performs data enhancement on frame-level samples to generate enhanced view pairs.Secondly, the enhanced view pairs are spliced to input a deep fake detection model to obtain frame-level features fused and enhanced.Thirdly, intra-class contrast learning loss, time sequence consistency constraint loss and frame weight loss are constructed to jointly optimize and train the deep fake detection model based on the frame-level features fused and enhanced.Finally, a video to be detected is input into the deep fake detection model to output frame-level confidence, and it is determined whether the video is a fake video, and frame-level fake positioning is realized.The application effectively realizes video-level detection and frame-level positioning, simultaneously alleviates the influence of label noise in part of fake videos, and improves the generalization and robustness of the system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly to a deep fake face video frame-level positioning method and system based on weakly supervised learning. BACKGROUND

[0002] With the rapid development of generative artificial intelligence (AIGC), various types of artificial intelligence generated content have been widely disseminated in social networks, film and television production and entertainment applications, and have shown important application value in these fields. Among them, deep fake, as a typical sub-type of AIGC, is often used for telecom fraud, false information dissemination, reputation defamation and interference with judicial evidence judgment, which poses a serious threat to public safety, personal privacy and the credibility of digital media. With the continuous progress of technology, attackers can even tamper with only part of the key frames in the video, thereby improving the concealment of the forgery and significantly increasing the detection difficulty. This type of partial tampering often introduces label noise during model training, and the video may also undergo compression, cropping, scaling and other post-processing operations during transmission, further reducing the detection accuracy. At the same time, obtaining accurate frame-level artificial annotation not only has high cost, but also is difficult to cover diversified forgery types, making the adaptability of existing supervised detection methods limited in real complex scenarios.

[0003] Existing deep fake detection methods can be mainly divided into supervised, unsupervised and weakly supervised methods. Supervised methods rely on a large amount of high-quality frame-level labeled data and can achieve high detection accuracy on known forgery types, but perform poorly on partially tampered videos and new types of forgeries, and the frame-level labeling cost is extremely high, limiting the practical application of the method. Unsupervised methods use frequency domain features, pixel consistency or artifact analysis to distinguish between true and false, reducing the dependence on labeling, but it is difficult to capture deep semantic differences and lacks robustness for complex forgeries or videos that have undergone compression, cropping, scaling and other post-processing. Existing weakly supervised deep fake detection methods usually use video-level labels for training, but mostly stay at the video overall level and fail to fully exploit the semantic differences between frames, making it difficult to effectively convert from video-level supervision to frame-level detection and positioning. In partially tampered videos, real frames and fake frames coexist, and direct use of video-level labels will cause the supervision signal to deviate, weakening the model's ability to recognize key fake frames. At the same time, videos that have undergone compression, cropping and other post-processing operations will further affect the inter-frame feature distribution, making the detection robustness and accuracy decline.

[0004] Therefore, the existing methods have the following limitations: first, there is a lack of frame-level learning mechanism for video-level labels, which cannot realize frame-level detection and forgery positioning under weak supervision conditions; second, the inter-frame semantic differences are not fully utilized, making it difficult to accurately identify the forgery features of partially tampered frames; third, direct training using video-level labels is easily disturbed by real frames, causing supervision noise and feature confusion; fourth, under complex post-processing conditions, the model lacks generalization and robustness, limiting the actual application effect. SUMMARY

[0005] The present application aims to solve the problem that existing deep forgery detection methods cannot achieve accurate frame-level detection and positioning under video-level label conditions. A deep forgery face video frame-level positioning method and system based on weak supervision learning are proposed. This method constructs a detection framework combining frame-level dynamic weighting and intra-class contrast learning, realizes frame-level detection and positioning under the condition of only having video-level labels, and realizes the mapping from video-level labels to frame-level detection and positioning through the design of a multi-example structure that supervises the video as a whole and analyzes the frame as the basic unit. In combination with intra-class contrast learning and dynamic weighting mechanism, the inter-frame heterogeneity is effectively modeled, the supervision bias is suppressed, and the robustness and detection accuracy of the model in complex scenarios are effectively improved. During the training process, first, the input video frame samples are subjected to various data augmentation, including noise addition, compression, rotation, and scaling, to generate semantically consistent augmented view pairs, improving the model's adaptability to post-processing forgery and diversified distortion. Then, frame-level feature extraction is performed on the augmented views, and temporal feature fusion is performed based on adjacent frame residuals to explicitly model the inter-frame motion discontinuity and forgery mutation features, and generate fused and enhanced frame-level features through residual perception gating and attention modules. Subsequently, intra-class contrast learning constraints are used to make features of the same class of frame samples aggregate in the feature space, constructing a stable and compact real sample feature space, providing reliable semantic anchors for the discrimination and positioning of forged frames. At the same time, temporal consistency constraint loss is introduced to constrain the smoothness between adjacent frame features, so that the model maintains a stable response to inter-frame natural changes in real videos, while being sensitive to forged mutation frames, thereby further improving the accuracy and robustness of frame-level detection. Finally, a frame-level dynamic weighting mechanism is introduced to adaptively adjust the weight based on the similarity between frame features and real clustering prototypes, to mine frame-level forgery clues from video-level labels, and realize accurate detection and positioning of partially tampered frames. Under the weak supervision condition of lacking accurate frame-level labels, the present application realizes the effective conversion from video-level supervision to frame-level detection, significantly improves the robustness and generalization ability of the deep forgery detection model, effectively balances the detection performance and labeling cost, and meets the efficient detection needs in complex real application scenarios.

[0006] To solve the above technical problems, the technical scheme of the present application is as follows:

[0007] A deep fake face video frame-level positioning method based on weakly supervised learning, comprising the following steps:

[0008] Step 1, construct a training set sample, the training set is constructed in units of video, the original video containing a face (including real video and fake video) is cropped into frame-level samples, each video corresponds to a "package", and each frame sample in the package corresponds to an "example". Among them, all frames in the real video package are real frames, and are uniformly labeled as real labels; the frame samples in the fake video package contain both tampered fake frames and un-tampered real frames, but since only video-level labels are used for training, all frames in the package need to be labeled as fake labels, thereby introducing label noise in the training set.

[0009] On this basis, data augmentation is performed on the frame-level samples to construct a multi-view sample pair for contrastive learning. Specifically, two different data augmentation methods are randomly applied to each frame-level example to generate view pairs with different appearances but consistent semantics. This enhancement process can improve the discriminability and robustness of the model in the feature space, providing diverse inputs and feature consistency constraints for subsequent residual feature enhancement and temporal modeling;

[0010] Step 2, the enhanced view pairs generated in step 1 are spliced to input the backbone neural network of the deep fake detection model for frame-level feature extraction. Then, the residual information between adjacent frame features is obtained by subtraction to explicitly model the motion discontinuity and fake mutation features between frames. Next, the residual information and the frame-level features are spliced in the channel dimension to input the residual-aware gating and attention fusion module to generate fused and enhanced frame-level features. This process preserves the spatial appearance features of the frames while enhancing the model's attention and discrimination ability for fake mutation frames.

[0011] Step 3, the fused and enhanced features obtained in step 2 are used to train the deep fake detection model. For the extracted fused and enhanced frame-level features, an intra-class contrastive learning loss is used to pull closer the features of frames of the same class (real frame-real frame), while not forcing the pull closer of frames of different classes (real frame-fake frame) or uncertain classes, thereby forming a stable and compact real sample feature space.

[0012] On this basis, a temporal consistency constraint loss is introduced to suppress the model's excessive response to frame mutations in real videos by calculating the smoothness loss between adjacent frame features, and to enhance its sensitivity to fake segment mutation frames. Finally, a frame weight loss is designed to dynamically assign frame weights based on the cosine similarity of the prototype features obtained by clustering frames corresponding to fake labels and real samples, and the weights are adjusted with the similarity to amplify the contribution of key fake frames in loss calculation.

[0013] The intra-class contrast learning loss, the temporal consistency constraint loss, and the frame weight loss are jointly optimized, and the backbone network is trained through back propagation to effectively suppress the interference of noise frames under weak supervision and improve the discrimination ability and positioning accuracy of the model for the partial fake frames.

[0014] Step 4, frame-level detection result output and fake positioning. After the training is completed, the video to be detected is input into the deep fake detection model to output frame-level confidence. The backbone network is used to extract the feature representation of each frame of the video, and the classification head outputs the frame-level confidence according to the fused and enhanced frame-level features. As long as there is a frame with a confidence greater than a threshold, the video is determined to be fake. On this basis, the frame-level confidence is arranged in the order of video frames, and a frame sequence interval with a confidence continuously higher than the threshold is identified, which corresponds to a suspected fake segment and is represented by the start and end frame numbers, thereby realizing frame-level fake positioning. Finally, the system returns the detection result to the user, including the video authenticity determination label, the fake frame sequence interval, and the frame-level confidence.

[0015] Preferably, the backbone neural network adopts a Vision Transformer structure, and a contrast learning loss function, a temporal consistency loss, and a weight loss function are introduced as constraints during the training process.

[0016] Preferably, the real sample clustering adopts a Gaussian mixture model based on cosine similarity for clustering to suppress the feature deviation of video frames caused by low-level factors such as illumination changes and skin color differences. The cluster centers generated by clustering are used as real sample prototypes to construct stable reference feature representations, so as to dynamically evaluate the similarity between fake frames and real frames during the training process, thereby realizing frame-level weight distribution and fake detection optimization under weak supervision.

[0017] Preferably, the aggregation method of the video-level fake confidence is to take the maximum value in the frame-level confidence, that is, the frame with the highest confidence in the video is taken as the video-level fake confidence, so as to realize sensitive detection of partial fake videos in accordance with the principle that “as long as there is a fake frame, the video is determined to be fake”.

[0018] In another aspect, the present application provides a deep fake face video frame-level positioning system based on weakly supervised learning, comprising the following modules:

[0019] The training set construction module is used to construct a training set in units of videos, split the videos into frame-level samples, and organize them into a “bag-instance” data structure required for multiple-instance learning. For each frame sample, data enhancement operations are performed, including noise disturbance, image compression, rotation, scaling, etc., to generate multiple view enhanced frame sample pairs. This module provides a basic sample and feature consistency constraint, providing input for subsequent frame feature extraction and weakly supervised training.

[0020] The feature extraction and fusion module is used for inputting the enhanced view generated in step 1 into a deepfake detection model, and extracting frame-level features through a backbone network. Subsequently, residual information obtained by differencing the frame-level features and adjacent frame features is subjected to attention fusion and gate operation to generate fused and enhanced frame-level features.

[0021] The joint optimization training module is used for constructing multiple loss functions based on the fused and enhanced frame-level features and performing joint optimization training. The module calculates an intra-class contrast learning loss for narrowing the features of frames of the same class, calculates a temporal consistency constraint loss for weighting the differences between continuous frame features through a sliding window and residual information, and calculates a frame weight loss for dynamically assigning frame weights according to the similarity between fake frames and real sample clustering centers. The three types of losses form a weighted total loss function for end-to-end training of the deepfake detection model.

[0022] The fake detection and positioning module is used for performing inference on an input video after training and outputting the fake confidence of each frame. According to the frame-level confidence, it is determined whether the whole video is a fake video, and the positioning of the frame-level fake area is realized to support the structured display of the fake frame distribution.

[0023] The present application has the following characteristics and beneficial effects:

[0024] (1) A deepfake detection and positioning learning framework based on weak supervision is proposed, which can realize frame-level true-fake detection and fake segment positioning only relying on video-level labels, breaking through the limitation of traditional methods that can only perform video-level determination, and significantly improving the detection capability of partial fake videos.

[0025] (2) A residual perception mechanism is introduced, and temporal residual information is constructed by the difference between adjacent frame features to guide the model to focus on the inter-frame mutation area, highlight the key fake area, and improve the response sensitivity of the model to fake frames.

[0026] (3) A residual-driven k-frame sliding window smoothing constraint is used to calculate a weighted temporal consistency loss for the fused features of continuous k frames, amplify the gradient response of fake mutation frames through a dynamic weighting function of residual perception, and suppress the influence of natural changes between real frames, so that the model can learn the temporal features and mutation patterns of fake videos, thereby further improving the positioning accuracy of key frames.

[0027] (4) A dynamic frame weight distribution strategy combined with a real sample prototype is used to adaptively adjust the frame contribution degree according to the feature similarity between the frame and the prototype, effectively alleviate the noise label problem of partial fake videos under weak supervision, and improve the detection robustness and generalization ability.

[0028] (5) The system design covers the whole process of "preprocessing, feature extraction, weight distribution, aggregation and positioning", supports dynamic iteration of modules, can adapt to new types of fake methods, and realizes high robustness and wide applicability. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0030] Figure 1 A flowchart of a weakly supervised detection and positioning model in an embodiment of the present application;

[0031] Figure 2 A flowchart of video detection in an embodiment of the present application. DETAILED DESCRIPTION

[0032] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0033] The present application proposes a multi-instance learning weak supervision framework, which regards video as "package" and frame as "instance", and only relies on video-level label for training to realize the authenticity detection and key frame positioning of each frame in the video. In this process, by combining the real sample prototype to dynamically distribute frame weight, the label noise problem existing in part of the fake video is effectively alleviated, so as to improve the robustness and generalization ability of the model in the frame-level detection and positioning task.

[0034] A "data enhancement and contrast learning" feature extraction mechanism is constructed, which simulates the enhancement mode of post-processing and contrast loss, so that the model learns robust frame features and improves the resistance ability of the model to post-processing interference.

[0035] Real sample clustering and weighted loss are integrated into training, the frame weight is adjusted by taking the clustering prototype as a reference, and then the video-level confidence is aggregated by weighted average, so that the system adapts to the real detection scene, and the overall robustness and practicability are improved.

[0036] In one aspect, the present application provides a deep fake face video frame-level positioning method based on weakly supervised learning, comprising the following steps:

[0037] Step 1, constructing a training set in units of video, and performing data enhancement on frame-level samples to generate enhanced view pairs.

[0038] Step 2, inputting the enhanced view pairs into a deep fake detection model, and obtaining frame-level features generated by fusion enhancement.

[0039] Step 3, based on the fusion enhanced frame-level features, construct intra-class contrast learning loss, temporal consistency constraint loss and frame weight loss, jointly optimize the training of deep fake detection model.

[0040] Step 4, after training, input the video to be detected into the deep fake detection model to output frame-level confidence, determine whether the video is a fake video, and realize frame-level fake positioning.

[0041] As preferred, in step 1, the original video dataset is obtained , wherein represents the original video, and respectively represents the real and fake labels, represents the number of videos; the face of the video is cropped by Dlib to obtain a frame-level image dataset , wherein represents the cropped video frame, represents the number of video frames, the label is inherited from the video-level label but may contain label noise in the fake video.

[0042] The frame-level samples are data enhanced , wherein includes noise, compression, flip, rotation and scaling, etc.; two enhanced views are randomly selected for each sample to generate a view pair to construct the final frame-level training set.

[0043] As preferred, in step 2, the view pair is input to the backbone neural network to extract high-dimensional feature representation , and mapped to low-dimensional feature representation , i.e. frame-level feature representation, through the projection head.

[0044] Further, in order to capture the dynamic change features between video frames, the temporal residual information is calculated for the features of adjacent frames , , ,

[0045]

[0046] The residual information and the view features of the corresponding frames generate the final fusion features through residual perception gating and attention mechanism:

[0047]

[0048] ​

[0049] wherein, denotes the residual absolute value, used to reflect the mutation intensity; is a splicing vector of the residual and the frame-level feature, and are the weight matrix and the bias of the gating layer, respectively; sigmoid function, used to limit the gating coefficient in the interval ; denotes the attention fusion operation based on residual guidance, which uses residual information to adjust the time-sensitive response of the view feature; the final fusion enhanced feature is controlled by the gating coefficient to weight the combination of residual information and the original view feature, thereby achieving dynamic perception and enhancement of key frame fake mutations.

[0050] As a preferred, the step 3 constructs a similarity matrix for all fusion enhanced features , specifically: taking the training batch as the unit, assuming that the batch contains samples, and the fusion enhanced feature of each sample is ( is the feature dimension), which constitutes a feature matrix . The cosine similarity of all sample pairs in the batch is calculated, and for the th feature and the th feature , the cosine similarity is , and the cosine similarities of all sample pairs are arranged to form a similarity matrix of , and all elements in the similarity matrix are divided by the temperature coefficient to obtain the scaled similarity matrix , and the element is .

[0051] At the same time, a mask matrix is constructed according to the sample label . When the sample belongs to the same class as the sample , the corresponding position element of the mask matrix is 1, otherwise it is 0, so as to screen the same class sample features. A contrast learning loss function is used to constrain the model training. The contrast learning loss function is defined as follows:

[0052]

[0053] wherein, is the number of samples in the current training batch; is the mask matrix; denotes the number of same-class samples corresponding to each sample . is the scaled similarity matrix element; the denominator is the sum of the exponential similarity of all other samples (excluding itself) in the batch. The above constraints achieve the aggregation of similar features and the separation of dissimilar features, thereby obtaining a robust frame-level feature representation.

[0054] As preferred, in step 3, the frame-level examples in the training set are processed, and first, the Gaussian mixture model based on cosine similarity is used for single-class clustering of the video frame features corresponding to the true label, so as to suppress the feature deviation of the video frame caused by low-level factors such as illumination change and skin color difference, and the clustering center is selected as the true sample prototype

[0055]

[0056] wherein represents the true label set of all true samples, is used as the reference feature of the true sample.

[0057] Subsequently, the cosine similarity between the frame features corresponding to the fake label and the true sample prototype is calculated as the weight:

[0058]

[0059] wherein, represents the frame weight, which is fixed as for the frame corresponding to the true label, and the frame weight corresponding to the fake label is determined by the cosine similarity, which is used to reflect the contribution of the frame to the video-level label.

[0060] When calculating the loss based on the weight , first, the frame-level feature weight is weighted according to , the weighted cosine similarity of the sample pair to the enhanced frame-level feature is calculated , and a weighted similarity matrix is constructed based on this, and the weight loss is calculated through the matrix

[0061]

[0062] wherein, is the focal term, so that the difficult samples with low "confidence" of the model contribute more.

[0063] Further, in order to suppress the excessive response of the model to the natural inter-frame difference in the real video, while amplifying the mutation features of the fake segment, for the features of consecutive frames and the adjacent frame residual information , the timing consistency loss is calculated:

[0064] ​

[0065] wherein, is the number of video frames, is the L2 norm, is a residual-aware dynamic weighting function to amplify the gradient response of abrupt frames:

[0066]

[0067] wherein, is the L2 norm of the neighboring frame residual information measuring the abruptness; is the maximum value of all neighboring frame residuals in the current video sequence, used for normalization to ensure the weighting value is within a reasonable range; is a hyper-parameter to control the amplification strength of residual magnitude on the weighting coefficient. When the residual of a certain frame is large (possibly a fake abrupt frame), its normalized value is high, greater than 1, making the frame contribute more in the temporal smoothing loss or gradient; when the residual is small (smooth frame or real frame), the weighting coefficient is close to 1, avoiding weakening the original gradient, thus maintaining the smoothness of the real frame.

[0068] In the training phase, the intra-class contrast loss, weight loss and temporal consistency loss are combined to form a total loss function:

[0069]

[0070] wherein is the intra-class contrast loss, is the weight loss, is the temporal consistency loss, , and is a hyper-parameter to adjust the relative importance of the three.

[0071] As a preferred, in step 4, the video to be detected , its frame set is input into the trained detection model to extract frame-level features and calculate the fake confidence of each frame , wherein . Subsequently, based on the frame-level confidence, video-level judgment is made: when any frame in the video is higher than the threshold , the entire video is determined to be fake, otherwise it is determined to be real, obtaining the video-level label :

[0072]

[0073] wherein Indicating the video as fake, Indicating the video as real. This strategy ensures that even if only part of the frames of the video are tampered with, the detection model can still judge it as a fake video, thereby improving the sensitivity and detection accuracy of partial fake videos.

[0074] On this basis, the frame-level confidence According to the time sequence, the frame sequence interval with confidence continuously higher than the threshold can be identified, which corresponds to the suspected fake segment and is represented by its start and end frame numbers, realizing frame-level fake positioning.

[0075] Embodiment:

[0076] The embodiment provides a deep fake face video frame-level positioning method based on weakly supervised learning, which is composed of two parts: model training stage and model detection stage.

[0077] (1) Model training stage: The main purpose of the model training stage is to build a detection model suitable for the weak supervision scene. By processing the noise labels in the partial fake video and simulating the post-processing interference in the video propagation, the model learns robust frame-level features and video-level decision logic, finally realizes accurate identification of deep fake videos in complex scenes, and avoids excessive dependence on high-quality frame-level labels, balances the labeling cost and detection performance, as shown in Figure 1 The specific training steps are as follows:

[0078] (1.1) Training set sample construction: The training set sample construction aims to convert the original video data into standardized data conforming to the multi-instance learning "bag-instance" framework, providing basic materials for model training, and clearly defining the association and potential noise problems of frame-level labels and video-level labels.

[0079] (1.1.1) First, obtain the original video dataset and determine the label rule, and construct the original video dataset wherein represents a single video sample, represents the number of videos, respectively represent the real and fake video labels, which will be used as the initial label basis for subsequent frame-level samples. It should be noted that some key frames in the fake video may be tampered with, resulting in noise in the subsequent frame-level labels.

[0080] (1.1.2) Then, frame extraction and face cropping are performed on the original video to construct a frame-level dataset. The original video is processed using the Dlib tool to first crop the frame-level images containing the face from the video, obtaining the frame-level image dataset wherein is a single frame-level image sample, represent the number of video frames, The label is inherited from the video level The label of the video But because the fake video is partially tampered, the frame-level label corresponding to the fake video May not be consistent with the authenticity of the actual frame, that is, there is label noise.

[0081] (1.1.3) Finally, the "bag-instance" structure is divided, and the multi-instance learning data format is clear. Each original video All frame-level images Corresponding to a "bag", a single frame-level image As an "instance", each "bag" corresponds to a unique video-level label , Forming the training data structure required by multi-instance learning, laying the foundation for subsequent model weakly supervised training based on video-level labels.

[0082] (1.2) Frame-level data augmentation and feature extraction: In order to enhance the adaptability of the model to the interference of video post-processing, apply enhancement functions , Including noise addition, compression, rotation and scaling operations, randomly select two enhancement methods to generate paired views , To simulate various disturbances in video propagation. Subsequently, the enhanced view pair is input into the backbone neural network , First extract high-dimensional features , Then map to low-dimensional features , Provide a basis for subsequent frame-level feature learning.

[0083] (1.3) In order to capture spatial appearance features and temporal change features at the same time, a residual-guided fusion mechanism is introduced. Calculate the residual information Between adjacent frame features , Used to highlight the mutation area of key frames. The residual information and the frame-level features are taken as two branches, and the final fusion enhanced frame-level features Are generated through residual-aware gating and attention fusion, realizing the enhanced perception of fake mutations and temporal discontinuity. Based on the fusion-enhanced frame-level features, construct the intra-class contrast loss , Make the same category frame features aggregate, form a stable and compact real sample feature space, provide a reliable reference for fake frame identification.

[0084] (1.4) Real sample clustering and dynamic frame weight allocation. Single-class clustering is performed on real frame features to obtain real sample prototypes ​A Gaussian mixture model based on cosine similarity is used to suppress feature shifts caused by low-level factors such as lighting and skin color. Subsequently, the cosine similarity between the features of the forged frame and the prototype is calculated to obtain adaptive frame weights. This reflects the contribution of each frame to the video-level tags, and yields... The weight of the real frame is fixed at 1. This stage serves as a bridge between the feature space and the frame-level weights, providing a basis for temporal constraints.

[0085] (1.5) Temporal Consistency Constraint. To amplify the features of fake mutations and suppress natural differences between real frames, a temporal consistency constraint is introduced. Continuous Frame fusion features Residual with adjacent frames Jointly calculate timing consistency loss This mechanism enables the model to capture fake mutation patterns in the time dimension while ensuring the smoothness of real frames, thereby improving the accuracy of keyframe localization.

[0086] (1.6) Finally, the intra-class contrast loss is... Frame weight loss With timing consistency loss A weighted average is used to form a total loss. Optimizing this total loss allows the model to learn robust frame-level features under weak supervision, while simultaneously ensuring effective forged frame identification and noise frame suppression, achieving high-precision frame-level detection and forged segment localization. The entire training process forms a logical closed loop: from video input to frame-level feature extraction, then to residual-aware fusion, weight allocation, and temporal constraints, ultimately achieving a comprehensive understanding and accurate detection of some forged videos through joint loss.

[0087] (2) Model Detection Stage: The main purpose of the model detection stage is to use the trained detection model to process the input video to be detected, extract features, calculate confidence, and aggregate the results to finally output accurate video-level forgery detection conclusions, which are then fed back to the user to judge the authenticity of the video, such as... Figure 2 As shown, the specific process is as follows:

[0088] (2.1) Video processing and confidence calculation: Video processing and confidence calculation involves converting the video to be detected into a format that the model can process, and obtaining frame-level and video-level forgery confidence through model inference, providing data support for the final judgment.

[0089] (2.1.1) First, acquire the video to be detected and extract frame-level samples. (Acquire the video to be detected) Following the same method as constructing the training set, frame-level images containing faces are extracted from the video to be detected to form a set of frames to be detected. (in The number of frames extracted from the video to be detected), ensure that the sample format of the input model is consistent with the training phase.

[0090] (2.1.2) Then, input the frame to be detected into the model, and calculate the frame-level fake confidence. Each frame in the set of frames to be detected Input the trained detection model , pass through the backbone neural network and classification head of the model, and calculate the fake confidence of the frame , wherein , The closer to 1 indicates the higher the possibility that the frame is fake, and the closer to 0 indicates the higher the possibility that the frame is real.

[0091] (2.1.3) Video-level fake judgment. Here, the "one frame fake, one video fake" strategy is adopted. Set the frame-level judgment threshold (typically 0.5), if there is any frame , the whole video is determined to be fake; otherwise, the video is determined to be real. This method can ensure that even if only the key frame is tampered with, the model can accurately identify the video fake.

[0092] (2.2) The output of the detection result aims to convert the video-level confidence into an intuitive detection conclusion, and clearly inform the user of the authenticity of the video to be detected, while providing frame-level positioning information to ensure the usability and clarity of the detection result.

[0093] (2.2.2) Frame-level fake segment positioning. Arrange the frame-level confidence in time sequence, identify the continuous or discrete frame sequence interval higher than the threshold as a suspected fake segment, and record it in the form of start and end frame numbers. Finally, the following information structure is returned to the user:

[0094]

[0095] Which includes video-level label , frame-level fake interval and confidence of each frame, so that the user can understand the overall state of the video and the specific fake frame position at the same time.

[0096] On the other hand, the present application provides a deep fake face video frame-level positioning system based on weakly supervised learning, which comprises the following modules:

[0097] The training set construction module is used to construct the training set in units of videos, and to perform data enhancement on the frame-level samples in the video to generate enhanced view pairs.

[0098] The feature extraction and fusion module is configured to input the enhanced view pair to the deep fake detection model, extract frame-level features through a backbone network, and fuse the frame-level features with residual information to obtain fused and enhanced frame-level features.

[0099] The joint optimization training module is configured to construct intra-class contrast learning loss, temporal consistency constraint loss, and frame weight loss based on the fused and enhanced frame-level features, and perform end-to-end training of the deep fake detection model using a total loss function formed by weighting the three losses.

[0100] The fake detection and positioning module is configured to, after training is completed, detect input videos by using the deep fake detection model, output frame-level confidence, determine whether the videos are fake videos according to the frame-level confidence, and realize positioning of frame-level fake regions.

[0101] To verify the effectiveness of the method in the weak supervision condition in training video-level labels to realize frame-level detection and positioning, ForgeryNet and FaceForensics++ (abbreviated as FF++) are selected as evaluation data sets. ForgeryNet is the largest DeepFake data set at present, supports four tasks of temporal fake positioning, spatial fake positioning, image fake classification, and video fake classification, contains 2.9 million images and 221,247 videos, and covers 15 different fake methods. Only part of the frames in the fake videos of the data set are tampered, which is highly consistent with the experimental scene of the present application. All fake videos are derived from four mainstream face data sets of CREMA-D, RAVDESS, VoxCeleb2, and AVSpeech. In order to strictly control the experimental conditions and avoid potential field bias, the real videos and the corresponding fake videos are used for training and testing on ForgeryNet, the face regions in the video frames are extracted by using the dlib tool, and the frames are divided into real frames and fake frames according to the frame-level labels provided by the data set, which are used for quantitative exploration of the influence of the proportion of part of the fake frames on the performance of the method. In the training stage, only video-level labels are used, and frame-level labels are not used. To simulate the scene of part of the fake videos, a noisy data set is designed, that is, a certain proportion of real frames are mixed in the fake samples (for example, 2 real frames are added in every 10 frames, corresponding to a noise rate of 20%), and the real samples remain pure. The FF++ data set is derived from 1,000 high-quality videos in YouTube-8M, and is one of the most widely used data sets in the DeepFake detection evaluation task. In the experiment, the standard data division and labeling are directly used for training and testing, and part of the real videos are reserved as reliable samples to ensure that there is no overlap between the real frames and the mixed fake frames in the training process.

[0102] In the experiment, the visual migration network VisionTransformer is used as the backbone neural network, all input images are uniformly adjusted to 299*299 size; the optimizer uses Adam, the learning rate is 1e-3, the weight decay is 1e-4; the batch size is 128, the temperature coefficient τ is set to 0.07; the performance evaluation index uses the area under the curve (AUC); the experiment uses NVIDIA3090TiGPU to ensure efficient computing performance. Through the above experimental settings, the effectiveness and superiority of the method in the video level label training to realize the frame level detection and positioning can be verified, and the influence of the proportion of the part of the fake frames on the robustness and detection performance of the model is quantitatively analyzed. It is worth noting that all experiments use the same hyperparameters under different noise rates, which conforms to the characteristics that the proportion of the part of the tampered video in the actual scene is unknown.

[0103] Table 1. AUC (%) comparison of ForgeryNet dataset under different noise rates of video level label

[0104]

[0105] Table 2. AUC (%) comparison of FF++ dataset under different noise rates of video level label

[0106]

[0107] From table 1 and table 2, it can be observed that under the noise ratio of 20%, the video detection algorithm EfficientNetB4 and Xception can still maintain certain performance in the video level DeepFake detection task, but as the noise ratio rises to 50% and above, its detection ability decreases significantly, especially under high noise conditions, it is almost impossible to effectively distinguish between real and fake videos.

[0108] In contrast, the method can maintain stable and excellent detection performance under different proportions of part of the fake frames (20%~80%), even under the condition of 80% high noise, it can still maintain a high AUC index. The experimental results show that the method can effectively learn the heterogeneity features of video frames from noisy training data by weakly supervised video level label training combined with multi-instance learning and dynamic frame weight distribution, and realize accurate distinction between real frames and fake frames. The design ensures that the model can still reliably output video level detection results in the part of the fake video scene, while supporting frame level positioning function, realizing accurate identification and labeling of the fake segment, fully embodying the robustness and generalization ability of the method under weak supervision.

[0109] The embodiments of the present application are described in detail above with reference to the accompanying drawings, but the present application is not limited to the described embodiments. Various changes, modifications, replacements, and variations of the embodiments including components can be made by those skilled in the art without departing from the principles and spirit of the present application, and still fall within the scope of the present application.

Claims

1. A frame-level localization method for deepfake face videos based on weakly supervised learning, characterized in that, Includes the following steps: Step 1: Construct a training set based on videos, and perform data augmentation on frame-level samples to generate augmented view pairs; The specific implementation of constructing a training set based on videos is as follows: a training set is constructed based on videos, and the original videos containing faces are cropped into frame-level samples. The original videos include real videos and fake videos. Each video corresponds to a package, and each frame sample in the package corresponds to an example. All frames in the real video package are real frames and are uniformly labeled as real labels. The frame samples in the fake video packet contain both tampered fake frames and undisturbed real frames. All frames in the packet are uniformly marked as fake. Step 2: Concatenate the enhanced view with the input depth forgery detection model and obtain the generated fused and enhanced frame-level features; Step 3: Based on the fused and enhanced frame-level features, construct intra-class contrastive learning loss, temporal consistency constraint loss, and frame weight loss, and jointly optimize and train the deep forgery detection model; The frame weight loss is based on the cosine similarity of the prototype features obtained by clustering the frames corresponding to the fake labels with the real samples. The cosine similarity is used as the frame weight for intra-class contrast learning loss weighting. Step 4: After training is complete, input the video to be detected into the deep fake detection model to output frame-level confidence, determine whether the video is fake, and achieve frame-level fake location.

2. The deep fake face video frame-level localization method based on weakly supervised learning according to claim 1, characterized in that, The specific implementation of data augmentation for frame-level samples is as follows: two different data augmentation methods are randomly applied to each frame-level example to generate view pairs that differ in appearance but are semantically consistent.

3. The deep fake face video frame-level localization method based on weakly supervised learning according to claim 2, characterized in that, Step 2 is specifically implemented as follows: extracting feature representations from the backbone neural network of the stitched input depth forgery detection model based on the enhanced view. Then, frame-level features are obtained through projection head mapping. Then, the residual information is obtained by subtracting the frame-level features of adjacent frames. Next, the residual information and frame-level features are concatenated along the channel dimension and then input into the gating and attention fusion module to generate fused and enhanced frame-level features.

4. The deep fake face video frame-level localization method based on weakly supervised learning according to claim 3, characterized in that, The gating and attention fusion module is specifically implemented by: processing residual information... Take absolute value and frame-level features The frames are concatenated, and then a gated vector is generated through linear activation mapping. Frame-level features are concatenated and then a gated vector is generated through linear activation mapping; residual information is then processed. After being concatenated with frame-level features, attention features are obtained through attention fusion and weighted by gating vector input. The gating vector is then inverted and weighted with the frame-level features. Finally, the attention features are added element-wise to obtain the final fused and enhanced frame-level features.

5. The deep fake face video frame-level localization method based on weakly supervised learning according to claim 4, characterized in that, In step 3, the fused and enhanced frame-level features obtained in step 2 are used to train the deep forgery detection model. For the extracted fused and enhanced frame-level features, intra-class contrastive learning loss is used to bring the features of frames of the same class closer together, while frames of different classes or uncertain classes are not forced to be brought closer together, thus forming a real sample feature space. Based on this, a temporal consistency constraint loss is introduced, which suppresses the over-response to inter-frame abrupt changes in real videos by calculating the smoothness loss between features of adjacent frames. The intra-class contrastive learning loss, temporal consistency constraint loss, and frame weight loss are jointly optimized by weighting, and the backbone network is trained by backpropagation. The backbone neural network adopts the VisionTransformer structure.

6. The deep fake face video frame-level localization method based on weakly supervised learning according to claim 5, characterized in that, The intra-class contrastive learning loss, temporal consistency constraint loss, and frame weight loss are specifically implemented as follows: Intra-class contrastive learning loss: Based on training batches, assuming each batch contains... Each sample has a fused enhancement feature. , The feature matrix is ​​constructed using the feature dimension. ; Calculate the cosine similarity of all sample pairs within a batch, for the . Features and the Features The cosine similarity is Arrange the cosine similarity of all sample pairs to form Similarity matrix Then divide all elements in the similarity matrix by the temperature coefficient. The scaled similarity matrix is ​​obtained. Element is Simultaneously, a mask matrix is ​​constructed based on the sample labels. When sample With sample When they belong to the same category, the corresponding position element of the mask matrix The value is 1 if the mask is set to 1 otherwise 0, thus filtering samples of the same type based on the mask matrix. and similarity matrix Calculate the intra-class contrastive learning loss function; Frame weight loss: The frame-level examples in the training set are processed. First, the frame-level features corresponding to the real labels are fused and enhanced, and then a Gaussian mixture model based on cosine similarity is used to perform single-class clustering. The cluster center is selected as the prototype of the real sample. Subsequently, the cosine similarity between the fused and enhanced frame-level features corresponding to the fake label and the real sample prototype is calculated as a weight. , Represents the frame weight for the frame corresponding to the true label. The frame weight corresponding to the forged label is fixed at 1, and is determined by the cosine similarity; based on the frame-level feature weights... and We perform weighted calculations, calculate the weighted cosine similarity of samples to the fused and enhanced frame-level features, construct a weighted similarity matrix based on this, perform weighted intra-class contrastive learning loss calculations, and obtain the frame weight loss. Temporal consistency constraint loss: Extract the fused enhancement features of k consecutive frames from the video and the residual information of adjacent frames, calculate the Euclidean distance, and use the residual-aware dynamic weighting function to perform weighted summation of the Euclidean distance; traverse all windows and combine the frame weights to calculate the average value to form the final temporal consistency loss; The window consists of k consecutive frames, all of which are obtained by sliding capture.

7. The deep fake face video frame-level localization method based on weakly supervised learning according to claim 6, characterized in that, Step 4 is specifically implemented as follows: After training, the backbone network is used to extract the fused and enhanced frame-level features of each frame of the video. The classification head outputs the frame-level confidence score based on the fused and enhanced frame-level features, and calculates the video-level forgery confidence score by aggregating the frame-level confidence scores. The video is determined to be forged if it is not, otherwise it is determined to be real. On this basis, the frame-level confidence scores are arranged in the order of the video frames, and the frame sequence intervals with confidence scores continuously higher than the threshold are identified. The frame sequence intervals correspond to the forged segments and are represented by their start and end frame numbers to achieve frame-level forgery localization. Finally, the detection results are returned to the user, including the video-level forgery confidence score, the video authenticity determination label, and the forged frame sequence interval.

8. The deep fake face video frame-level localization method based on weakly supervised learning according to claim 7, characterized in that, In step 4, the video to be detected is obtained. Input its frame set into the trained detection model Extract fused and enhanced frame-level features and calculate the forgery confidence score for each frame. ; Subsequently, video-level determination is performed based on frame-level confidence: when any frame in the video... Above the threshold If the video is fake, it is determined to be genuine; otherwise, it is determined to be real and a video-level tag is obtained. : Frame-level confidence Arranged by time series, it is possible to identify those with confidence levels continuously above a threshold. The frame sequence intervals, which correspond to suspected forged segments, are represented by their start and end frame numbers to achieve frame-level forgery location.

9. A deepfake face video frame-level localization system based on weakly supervised learning, used to implement the deepfake face video frame-level localization method according to any one of claims 1 to 8, characterized in that, Includes the following modules: The training set construction module is used to build training sets on a video-by-video basis and perform data augmentation on frame-level samples in the video to generate augmented view pairs. The feature extraction and fusion module is used to input the enhanced view pairs into the deep forgery detection model, extract frame-level features through the backbone network, and fuse them with residual information to obtain fused and enhanced frame-level features; The joint optimization training module, based on the fused and enhanced frame-level features, constructs intra-class contrastive learning loss, temporal consistency constraint loss, and frame weight loss, and uses the total loss function formed by the weighting of the three to train the deepfake detection model end-to-end; The forgery detection and localization module is used to detect input videos after the deep forgery detection model has been trained, output frame-level confidence scores, determine whether the video is a forgery based on the frame-level confidence scores, and locate the forgery region at the frame level.

Images