Multi-mode deep forgery continuous detection method based on playback

Abstract

The invention relates to a multi-mode deep counterfeiting continuous detection method based on playback. The method comprises the following steps: constructing a deep counterfeit video detection model; noise is added to historical multi-modal video data, the multi-modal video data added with the noise and the multi-modal video data are input into an empirical distribution stability estimation module for feature extraction and feature fusion, and a stable sample and a sensitive sample are selected based on corresponding feature distribution and stored in a memory pool for empirical playback; gradient-based disturbance is injected into a multi-layer network for a selected replay sample in a hierarchical disturbance replay enhancement module, and a current to-be-detected data sample and a sample in a memory pool are combined for model training in a deep counterfeiting detection continuous detection module; and detecting the multi-modal video data to be detected according to the optimized deep counterfeit video detection model. By adopting the method, the video counterfeiting detection accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of video deep fake detection, and in particular to a multimodal deep fake continuous detection method based on playback. Background Art

[0002] As deepfake technology evolves towards multimodality (such as audio and video fusion), video forgery detection, as its core scenario, is facing more complex challenges. Current single-modal detection methods (such as analyzing only the spatial / temporal features of videos) are difficult to deal with multimodal collaborative forgeries (such as simultaneous tampering of video face-swapping and audio synthesis), while traditional continuous learning technologies expose three major contradictions in video scenarios: First, there is a conflict between the dynamic transformation of feature space and the retention of historical knowledge. New video forgery technologies (such as expression transfer based on 3DMM and time-domain interpolation with optical flow compensation) will cause the model's extraction mechanism for historical features (such as the consistency of facial key points and the distribution of inter-frame motion vectors) to fail, resulting in significant drift in the feature space. Methods such as experience replay are unable to restore the joint distribution of multimodal data (such as only retaining video and ignoring audio associations), making it difficult to effectively alleviate catastrophic forgetting. Second, there is a contradiction between the characteristics of multimodal data and the efficiency of incremental learning. Video data itself has high storage costs (about 100MB for a single minute of 1080p video) and temporal dependencies. If the traditional replay method retains the full amount of historical videos, it will lead to storage explosion. If a single frame is randomly sampled, the temporal dynamic clues will be lost (such as abnormal blinking frequency and inconsistent head movement trajectory), making it difficult for the model to capture forged temporal consistency vulnerabilities. Third, there is a disconnect between cross-modal association modeling and continuous learning frameworks. Existing detection models mostly process each video modality independently (such as spatial texture and temporal optical flow), and lack modeling of the "video tampering-audio tampering" relationship (such as lip shape and speech synchronization). In incremental learning, new multimodal forgery combinations (such as face-changing + voice-changing) will destroy the cross-modal dependency knowledge that the model has learned (such as the mapping of facial action units and speech acoustic features), resulting in a sharp drop in joint detection performance.

[0003] Traditional single-modal forged data detection methods are difficult to effectively apply to the continuous detection of multimodal deep fakes, resulting in low accuracy in video forgery detection. Summary of the Invention

[0004] Based on this, it is necessary to provide a playback-based multimodal deep fake continuous detection method that can improve the accuracy of video forgery detection to address the above technical problems.

[0005] A multimodal deepfake continuous detection method based on playback, the method comprising:

[0006] Acquire historical multimodal video data; build a deep fake video detection model; the deep fake video detection model includes an empirical distribution stability estimation module, a hierarchical perturbation replay enhancement module, and a deep fake detection continuous detection module;

[0007] Noise is added to historical multimodal video data, and the noise-added multimodal video data and multimodal video data are input into the empirical distribution stability estimation module for feature extraction and feature fusion to obtain the corresponding feature distribution; based on the corresponding feature distribution, stable samples and sensitive samples are selected and stored in the memory pool for experience replay;

[0008] In the hierarchical perturbation replay enhancement module, the selected replay samples are injected with gradient-based perturbations in the multi-layer network to obtain enhanced replay samples and store them in the memory pool;

[0009] In the deepfake detection continuous detection module, the current data sample to be detected is combined with the samples in the memory pool for model training to obtain an optimized deepfake video detection model;

[0010] The multimodal video data to be detected is detected based on the optimized deep fake video detection model.

[0011] The above-mentioned multimodal deep fake continuous detection method based on replay, this application retains the existing multimodal forgery technology knowledge by replaying previously representative forged videos, and alleviates the catastrophic forgetting caused by learning new forged data. First, the empirical distribution stability estimation module is designed to select samples with stable transformation and sensitive samples by simulating the forged feature transformation of incremental training, while retaining the representativeness and boundary knowledge of the old forged data, thereby reducing the loss of existing forged information. Secondly, the constructed replay-enhanced hierarchical perturbation method adds perturbations to the feature layer of the empirical video samples, enriching the diversity of forgery technology and increasing the information breadth of the old forgery tasks, thereby improving the robustness and generalization of the deep fake video detection model, and then improving the accuracy of video forgery detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 1 is a flowchart of a playback-based multimodal deep fake continuous detection method in one embodiment;

[0013] Figure 2 Schematic diagram of a framework of a multimodal deep fake continuous detection method in one embodiment. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0015] In one embodiment, Figure 1 and Figure 2 As shown, a multimodal deep fake continuous detection method based on playback is provided, including the following steps:

[0016] Step 102: Acquire historical multimodal video data; construct a deep fake video detection model; the deep fake video detection model includes an empirical distribution stability estimation module, a hierarchical perturbation replay enhancement module, and a deep fake detection continuous detection module.

[0017] Step 104: add noise to the historical multimodal video data, input the noise-added multimodal video data and the multimodal video data into the empirical distribution stability estimation module for feature extraction and feature fusion to obtain the corresponding feature distribution; based on the corresponding feature distribution, select stable samples and sensitive samples and store them in the memory pool for experience replay.

[0018] An empirical distribution stability estimation module is designed to add noise to historical multimodal video data to simulate the forged feature transformations of incremental training. By adding Gaussian noise perturbations to old samples to evaluate their feature shifts, it is possible to identify stable samples (with small feature shifts, more representative of the old task) and sensitive samples (with large feature shifts, close to the old task classification boundaries). This allows the representativeness and boundary knowledge of old forged data to be retained simultaneously, effectively alleviating the significant drift in the feature space caused by new video forgery techniques, reducing the loss of existing forged information, and avoiding catastrophic forgetting caused by the failure of the feature extraction mechanism. This allows the model to retain and utilize historical knowledge when facing multimodal collaborative forgeries, improving detection accuracy.

[0019] A dual sample selection mechanism is also used to select samples that maintain stability and samples that indicate sensitivity. This not only preserves the representative features of the old task, but also takes into account the boundary knowledge of the old task. This enables the model to better utilize historically learned information when dealing with new multimodal forgeries, improving its ability to discriminate against forged videos and ultimately increasing detection accuracy.

[0020] In addition, the empirical distribution stability estimation module selects stable and sensitive samples and stores them in the memory pool for experience replay, rather than retaining the entire historical video (which would cause storage explosion) or randomly sampling single frames (which would lose temporal dynamic clues) as in traditional replay methods. This approach controls storage costs while retaining important temporal dynamic clues in multimodal video data (such as abnormal blinking frequency and inconsistent head movement trajectories), enabling the model to capture forged temporal consistency vulnerabilities, better handle the high storage cost and temporal dependency issues of multimodal data, improve the efficiency of incremental learning, and help improve detection accuracy.

[0021] In addition, the empirical distribution stability estimation module in the deep fake video detection model extracts and fuses features from the multimodal video data and multimodal video data after adding noise to obtain the corresponding feature distribution. This way of fusing multimodal features enables the model to comprehensively consider the different modal information of the video (such as spatial texture, temporal optical flow, etc.) and the correlation between "video tampering-audio tampering" (such as lip shape and speech synchronization), making up for the shortcomings of existing detection models that process each modality of the video independently and lack cross-modal correlation modeling. In incremental learning, by retaining and enhancing the feature diversity of old samples, the model can maintain the learned cross-modal dependency knowledge (such as the mapping of facial action units and speech acoustic features), avoiding the problem that new multimodal forgery combinations (such as face-changing + voice-changing) destroy this knowledge and cause a sharp drop in joint detection performance, thereby improving the model's joint detection performance for multimodal deep fakes and improving the accuracy of video forgery detection.

[0022] In step 106 , the hierarchical perturbation replay enhancement module injects gradient-based perturbations into the multi-layer network to the selected replay samples, thereby obtaining enhanced replay samples and storing them in a memory pool.

[0023] The Hierarchical Perturbation Replay Augmentation module injects gradient-based perturbations into the multi-layer network to select replay samples, generating enhanced replay samples that are stored in a memory pool. This operation increases the feature diversity of the experience samples, providing more effective information to maintain previous task knowledge. This rich feature diversity enables the model to learn more comprehensive multimodal forgery features, thereby better handling different types of forgery and improving detection accuracy.

[0024] In step 106, the deep fake detection continuous detection module combines the current data sample to be detected with the samples in the memory pool for model training to obtain an optimized deep fake video detection model.

[0025] The deepfake detection continuous detection module combines the current data samples to be detected with the samples in the memory pool for model training to obtain an optimized deepfake video detection model. This training method that combines new samples with historical samples enables the model to continuously update and optimize its own parameters to adapt to new forged data and scenarios, further improving the robustness and generalization of the model, and ultimately improving the accuracy of video forgery detection.

[0026] Step 106: Detect the multimodal video data to be detected based on the optimized deep fake video detection model.

[0027] The innovation of this application lies in the optimization process of the model. The process of using the optimized model for detection is the same as the existing technology and will not be described in detail in this application.

[0028] The above-mentioned multimodal deep fake continuous detection method based on replay, this application retains the existing multimodal forgery technology knowledge by replaying previously representative forged videos, and alleviates the catastrophic forgetting caused by learning new forged data. First, the empirical distribution stability estimation module is designed to select samples with stable transformation and sensitive samples by simulating the forged feature transformation of incremental training, while retaining the representativeness and boundary knowledge of the old forged data, thereby reducing the loss of existing forged information. Secondly, the constructed replay-enhanced hierarchical perturbation method adds perturbations to the feature layer of the empirical video samples, enriching the diversity of forgery technology and increasing the information breadth of the old forgery tasks, thereby improving the robustness and generalization of the deep fake video detection model, and then improving the accuracy of video forgery detection.

[0029] In one embodiment, the multimodal video data after adding noise and the multimodal video data are input into the empirical distribution stability estimation module for feature extraction and feature fusion to obtain the corresponding feature distribution, including:

[0030] The multimodal video data after adding noise and the multimodal video data are input into the empirical distribution stability estimation module, and the video editor and audio editor are used to extract the modal stability features respectively, thereby obtaining the original audio features, the original video features, the noisy audio features and the noisy video features;

[0031] The original audio features, original video features, noisy audio features and noisy video features are fused according to the heterogeneous graph neural network to obtain the original feature distribution and the noise-based feature distribution.

[0032] In a specific embodiment, the multimodal deep fake video includes a visual modality v and an auditory modality a, and their corresponding labels y∈Y. Assume that the total number of samples in the training phase is N, and i∈N represents the i-th sample. In the continuous learning scenario, the feature drift caused by the transformation of the new task domain may cause the deep fake data of the previous task to forget knowledge. This application simulates the continuous learning scenario in the real world by adding noise to the original video. The introduction of noise will cause different degrees of offset in the original data, so that representative replay samples can be selected according to the degree of simulated feature offset. In order to distinguish between stable samples and sensitive samples, the input data includes the original video stream and the noise-based video stream.

[0033] In the feature extraction stage, the video encoder VisualEncoder and the audio encoder AudioEncoder are used to extract modality-specific features, respectively denoted as V i and A i In the noise-based video stream branch, the Gaussian noise function is defined as θ. In addition, the noise-based visual features and audio features are denoted as and

[0034] V i =VisualEncoder(v i ),A i =AudioEncoder(a i )

[0035] This application uses a heterogeneous graph neural network (GNN) to extract fusion features from two streams. Where L represents the number of layers in the backbone network, and l∈[1,L]. Among these layers, the first l-1 layers are graph attention layers (GAT), and the last layer is a linear classification layer (LC). l represents the function of the lth layer. Therefore, the original unknown type video (V i ,A i ) The output after the lth layer can be defined as z (i,l) , as shown in the following formula:

[0036]

[0037] In addition, noise-based video The output of layer l can be defined as The details are as follows:

[0038]

[0039] Through the hierarchical feature transformation of the network layer, discriminative feature representations can be extracted from the original video and the noise-based video.

[0040] In one embodiment, selecting stable samples and sensitive samples based on corresponding feature distributions and storing them in a memory pool for experience replay includes:

[0041] Based on the corresponding feature distribution, calculate the JS divergence between the feature distributions, use SortAsc to sort the elements in the JS divergence in ascending order, and obtain the sorted sequence;

[0042] The samples in the sorted sequence are divided into stable samples and sensitive samples according to their own indexes and stored in the memory pool for experience replay.

[0043] In a specific embodiment, z (i,l) and The final layer feature distributions of the original video and its noise-perturbed ones are considered respectively. Subsequently, the Jensen-Shannon (JS) divergence between the original feature distribution and the noise-based feature distribution is calculated. JS divergence provides a symmetrical, balanced, and stable metric for quantifying distribution differences, outperforming traditional methods in these aspects. The replay samples selected based on JS divergence are more representative. Larger JS values ​​indicate that the original sample is more sensitive to noise and are classified as sensitive samples; conversely, smaller JS values ​​reflect greater stability under noise interference and are classified as stable samples.

[0044] In one embodiment, calculating the JS divergence between feature distributions includes:

[0045] Calculate the JS divergence between feature distributions as

[0046]

[0047] in, It is z (i,L) and The average distribution of D KL represents the Kullback-Leibler divergence, z (i,l) and Represent the original feature distribution and the noise-based feature distribution respectively.

[0048] Among them, z and M i D between KL It is given by:

[0049]

[0050] In one embodiment, the stable sample is

[0051] CS k (S) = {s∈S|index(s) < k}

[0052] Where S represents the sorted sequence, k represents the index threshold, and index represents a function whose input is the element s in the sorted sequence S and whose output is the index value corresponding to the element s.

[0053] In one embodiment, the sensitive sample is

[0054] UCS k (S)={s∈S|index(s)≥N-k+1}

[0055] Where N represents the total number of samples of multimodal video data in the training phase.

[0056] In one embodiment, samples in a sorted sequence are divided into stable samples and sensitive samples according to their indexes and stored in a memory, including:

[0057] The samples in the sorted sequence are divided into stable samples and sensitive samples according to their own indexes and stored in memory The process is described as

[0058]

[0059] Where t and T represent the number of current tasks and total tasks respectively, and t∈[1,T] represents the memory state of the current task t. Indicates storage in memory A video sample from UCS k (S) indicates sensitive samples, CS k (S) indicates stable samples.

[0060] In one embodiment, the layer-level perturbation replay enhancement module injects gradient-based perturbations into a multi-layer network to select a replay sample to obtain an enhanced replay sample, including:

[0061] In the hierarchical perturbation replay enhancement module, the selected replay sample is injected into the multi-layer network based on the gradient perturbation, and the enhanced replay sample is obtained as

[0062]

[0063] Among them, z (j,l) and z l Respectively represent the memory The original and enhanced l-th layer features, and F l represents the concentration function after the lth layer, α represents the intensity of the perturbation factor, ||z (j,l) ||2 represents z (j,l) The L2 norm of .

[0064] In a specific embodiment, the layer-by-layer perturbation augmentation replay module is designed to inject perturbations into specific layers of the replay samples selected by the empirical distribution stability estimation module, thereby training and generating competitive enhanced features. These enhanced samples effectively push the replay samples away from the decision boundary, increasing feature diversity and providing richer information for preserving previous task knowledge.

[0065] To enhance the replay samples, the process first calculates the gradients of all layers in the model from the backpropagation process, which determines the direction and strength of the enhancement. Then, for each replay sample, the input of a layer is randomly selected and enhanced under the gradient perturbation corresponding to this layer. The specific enhancement process is shown in the following formula:

[0066]

[0067] where z (j,l) and z l Respectively represent the memory The original and enhanced l-th layer features, and F l represents the concentration function after the lth layer. Here, α represents the strength of the perturbation factor. By using the L2 norm, the input ||z (j,l) ||2 is designed to balance the different scales between different layers. The scale factor α is used as a hyperparameter to prevent adversarial overfitting. Therefore, the above equation performs a gradient ascent step on the selected layer l to transform the input feature z (j,l) Enhanced α||z (j,l) ||2 amplitude.

[0068] In one embodiment, the deepfake detection continuous detection module combines the current data sample to be detected with the samples in the memory pool to perform model training, including:

[0069] The current data sample to be detected (v j ,a j ,y j ) and samples stored in the memory pool Combined, a mixed batch is formed as the training set and the enhanced classification loss of the memory sample is expressed as The classification loss of new task samples is expressed as Construct loss function;

[0070] The model is trained based on the training set and loss function.

[0071] In one embodiment, the loss function is

[0072]

[0073] Where |B| represents the length of a batch.

[0074] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0075] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0076] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and such modifications and improvements are intended to fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A multimodal deepfake continuous detection method based on playback, characterized by: The method comprises: Acquire historical multimodal video data; construct a deep fake video detection model; the deep fake video detection model includes an empirical distribution stability estimation module, a hierarchical perturbation replay enhancement module, and a deep fake detection continuous detection module; Noise is added to the historical multimodal video data, and the noise-added multimodal video data and the multimodal video data are input into an empirical distribution stability estimation module for feature extraction and feature fusion to obtain a corresponding feature distribution; based on the corresponding feature distribution, stable samples and sensitive samples are selected and stored in a memory pool for experience replay; In the hierarchical perturbation replay enhancement module, the selected replay samples are injected with gradient-based perturbations in the multi-layer network to obtain enhanced replay samples and store them in the memory pool; The deep fake detection continuous detection module combines the current data sample to be detected with the samples in the memory pool for model training to obtain an optimized deep fake video detection model; The multimodal video data to be detected is detected based on the optimized deep fake video detection model.

2. The method according to claim 1, characterized in that The noise-added multimodal video data and the multimodal video data are input into the empirical distribution stability estimation module for feature extraction and feature fusion to obtain the corresponding feature distribution, including: The multimodal video data after adding noise and the multimodal video data are input into the empirical distribution stability estimation module, and the video editor and audio editor are used to extract the modal stability features respectively, thereby obtaining the original audio features, the original video features, the noisy audio features and the noisy video features; The original audio features, original video features, noisy audio features and noisy video features are fused according to a heterogeneous graph neural network to obtain an original feature distribution and a noise-based feature distribution.

3. The method according to claim 1, characterized in that Stable samples and sensitive samples are selected based on the corresponding feature distribution and stored in the memory pool for experience replay, including: Based on the corresponding feature distributions, the JS divergence between the feature distributions is calculated, and the elements in the JS divergence are sorted in ascending order using SortAsc to obtain a sorted sequence; The samples in the sorted sequence are divided into stable samples and sensitive samples according to their own indexes and stored in the memory pool for experience replay.

4. The method according to claim 3, characterized in that Calculating the JS divergence between the feature distributions includes: Calculate the JS divergence between the feature distributions as in, It is z (i,L) and The average distribution of D KL represents the Kullback-Leibler divergence, z (i,l) and Represent the original feature distribution and the noise-based feature distribution respectively.

5. The method according to claim 3, characterized in that The stable sample is CS k (S)={s∈S|index(s)<k} Where S represents the sorted sequence, k represents the index threshold, and index represents a function whose input is the element s in the sorted sequence S and whose output is the index value corresponding to the element s.

6. The method according to claim 4, characterized in that The sensitive samples are UCS k (S)={s∈S|index(s)≥N-k+1} Where N represents the total number of samples of multimodal video data in the training phase.

7. The method according to claim 3, characterized in that The samples in the sorted sequence are divided into stable samples and sensitive samples according to their own indexes and stored in memory, including: The samples in the sorted sequence are divided into stable samples and sensitive samples according to their own indexes and stored in memory The process is described as Where t and T represent the number of current tasks and total tasks respectively, and t∈[1,T] represents the memory state of the current task t. Indicates storage in memory A video sample from UCS k (S) indicates sensitive samples, CS k (S) indicates stable samples.

8. The method according to claim 1, characterized in that The layer-level perturbation replay enhancement module injects gradient-based perturbations into the multi-layer network to select the replay samples, thereby obtaining enhanced replay samples, including: In the hierarchical perturbation replay enhancement module, the selected replay sample is injected into the multi-layer network based on the gradient perturbation, and the enhanced replay sample is obtained as Among them, z (j,l) and z l Respectively represent the memory The original and enhanced l-th layer features, and F l represents the concentration function after the lth layer, α represents the intensity of the perturbation factor, ||z (j,l) ||2 represents z (j,l) The L2 norm of .

9. The method according to claim 1, characterized in that The deepfake detection continuous detection module combines the current data sample to be detected with the samples in the memory pool to perform model training, including: The current data sample to be detected (v j ,a j ,y j ) and samples stored in the memory pool Combine, form a mixed batch as the training set and express the enhanced classification loss of memory samples as l(F l (z l ),y j ), the classification loss of the new task sample is expressed as l(F0(v j ,a j ),y j ) Construct loss function; Model training is performed according to the training set and the loss function.

10. The method according to claim 9, characterized in that The loss function is Where |B| represents the length of a batch.

Images