A deepfake video detection method based on weighted feature pyramid

Abstract

The application discloses a kind of deep fake video detection methods based on weighted feature pyramid, belong to video identification technical field;Contain the following steps: construct fusion fake attention map's weighted bidirectional feature pyramid network, introduce the multi-task learning mechanism based on explicit attention mechanism;Training deep fake video detection network;Cut high-quality deep fake video in face information;The high-quality picture after processing is input to the deep fake video detection network after training and output whether belong to fake video.The application makes full use of the advantage that convolutional neural network shows in the image fake detail extraction in degree video fake detection, the fusion process of low layer and high layer feature map is supervised by fake attention map, while reducing information redundancy, the sensitivity of model to high-quality fake area is enhanced.Feature map is supervised by the generated fake position label, explicitly guide model to focus on sensitive area prone to artifact, improve the generalization ability of model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine vision and forgery detection, and in particular to a deepfake video detection method based on a weighted feature pyramid. BACKGROUND

[0002] With the rapid development of deep learning technology and generative adversarial networks (GAN), generative artificial intelligence has changed the production mode of network content, and deepfake technology has made significant progress, giving birth to Deepfakes, DeepFaceLab, Faceswap and other face-changing tools. These technologies and tools greatly improve the efficiency of content production, but also bring great challenges to network security risk management. Therefore, guiding and regulating the application of generative artificial intelligence technology, especially the detection technology for high-quality fake videos, and building an "AI vs. AI" closed-loop ecosystem have become the current research hotspot.

[0003] The current generative artificial intelligence generation technology is rapidly evolving, and the details of the generated content are increasingly realistic, with a continuously shortened update cycle. The generalization ability of the traditional forgery trace feature extraction method is difficult to cover emerging generation technologies, increasing the defense difficulty of forgery detection. Research shows that high-quality deepfake videos usually only have weak and difficult-to-detect artifacts in the local area of the forged image. Most of the existing deepfake detection methods use supervised binary classification models under implicit attention mechanisms, which can enable the model to learn the detailed feature information of the forged area spontaneously, but the stability across datasets is difficult to maintain.

[0004] Therefore, in order to improve the detection accuracy of high-fidelity forgery technology and take into account the strong model generalization, this paper proposes a deepfake video detection method based on a weighted feature pyramid. The network architecture used can fuse low-level feature maps and high-level feature maps with each other, and by introducing a forged attention map during the fusion process, it helps to implicitly detect detailed artifacts. At the same time, by using self-generated forgery data synthesis technology, explicit attention mechanisms are effectively integrated into the multi-task learning framework to deal with the forgery details generated by unknown forgery techniques and improve the model's generalization ability. SUMMARY

[0005] The purpose of the present application is to provide a deepfake video detection modeling method based on a weighted feature pyramid, which accurately captures global artifacts and detailed artifacts in forged images while maintaining strong generalization ability to unknown forgery means.

[0006] The technical solution adopted by the present application is a deepfake video detection method based on a weighted feature pyramid, which specifically includes the following steps:

[0007] Step 1: Constructing a spatial feature extraction network

[0008] EfficientNetV2 is used as the backbone to extract multi-scale features, and its output usually includes multiple down-sampled feature layers (C3, C4, C5), corresponding to down-sampling ratios of 8, 16, and 32. After C5, max pooling is used to generate P6 (stride = 64) and P7 (stride = 128) feature layers. The purpose of this is to better accommodate BiFPN. The spatial resolution of the five input levels (P3-P7) of BiFPN is p3=48×48; p4=24×24; p5=12×12; p6=6×6, p7=3×3. Each layer first uses a 1×1 convolution to reduce (or increase) the channel to 256 to unify the dimensions.

[0009] Further, a CBAM-based fake attention map is introduced. In CBAM, a hole convolution and a learnable mask are introduced, and the spatial and channel information of the fake area is weighted each time the features are fused. To fully exploit multi-scale information, the spatial attention module of FA-CBAM replaces the original 7×7 convolution with a 3×3 hole convolution, which enhances the receptive field without changing the parameters. At the same time, the supervision function of the multi-task learning branch is combined, the mask of the learnable mask branch is transmitted in reverse, an auxiliary loss is added to the output of the spatial attention of FA-CBAM, and it is required to align with the fake mask predicted by the mask branch. The channel number of the intermediate layer in the multi-layer perceptron (MLP) is reduced to 8, which reduces the parameter quantity while still retaining the channel feature information.

[0010] Step 2: Constructing a multi-task learning network based on explicit attention mechanism

[0011] Blending-based is used to generate pseudo-labels on the dataset and adjust the size. The picture is initialized to 0, and the value of the ROI box covered area is designed to 1, then down-sampling is performed, the purpose is to keep the size consistent and pixel-level alignment. The label (size 384×384) is obtained as the supervision label.

[0012] Further, a three-branch structure is designed: a classification branch, a learnable mask branch, and an enhanced self-consistency branch. The classification branch is used to judge the authenticity of the final output result. Because high-quality detection is required, the feature map should be as detailed as possible. Therefore, the multi-level output is subjected to a bilinear interpolation operation and up-sampled to the smallest receptive field feature map size. The multi-scale input layer of the above BiFPN - Since the resolution of is the largest, the As a baseline for the feature maps, other output layers are also adjusted to this baseline resolution.

[0013] Furthermore, a mask branch can be learned as an explicit training strategy, explicitly guiding the network to focus on specific feature regions through the learning of sensitive point labels. and As the input feature layer. Unify the fusion size, first... After upsampling by 2 times and The feature map F is obtained by concatenating the channels. For the concatenated F, multiple convolutional layers and upsampling are used to gradually restore it to the original image size. Since self-generated labels are used in the dataset processing stage, the feature map size needs to be changed back to the original image size to facilitate feature learning and explicit guidance for forged regions.

[0014] Furthermore, the enhanced self-consistency branch is used to guide the learnable mask branch, thereby enhancing generalization ability. Based on the feature map F of the learnable mask branch, another branch is copied, and Dropout is applied to it first, followed by LayerNorm, to obtain the perturbation feature F', which is used for the consistent supervised learning of F.

[0015] Step 3: Train and build a deepfake video detection model based on a weighted feature pyramid

[0016] The specific steps for training a deepfake video detection model based on a weighted feature pyramid are as follows:

[0017] The RetinaNet face extractor was used to extract faces from video frames. The input image was cropped and its resolution was adjusted to 384×384. The backbone network was EfficientNetV2-S pre-trained on ImageNet, with 7 layers. After passing through F-BiFPN, layers 4 and 5 were selected as the mid-to-high-level outputs. The steps for dividing the fake video dataset into training, validation, and test sets and standardizing it were as follows: the dataset was divided into training, validation, and test sets in a 3:1:1 ratio, and then rotated and scaled. The validation set was used to adjust hyperparameters, and finally, the model performance was evaluated using the test set.

[0018] Step 4: Input the processed video data into the trained weighted feature pyramid deep fake video detection network. The output of the network model is a probability, which shows the accuracy of the final result in judging whether the video is real or fake.

[0019] Compared with the prior art, the present invention has the following advantages:

[0020] The weak detail artifacts of high-quality forged videos are extracted, the response of the possible forged area in each layer feature is weighted and enhanced by introducing a forged attention map module, so that a multi-scale fusion feature map with semantic information and edge details is generated.

[0021] An explicit attention mechanism is introduced, and a learnable mask branch and an enhanced self-consistency branch are introduced on the basis of the traditional binary classification task, which are respectively used to capture the spatial position of the forged area and improve the robustness of the model to environmental disturbance and non-forged noise. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a deep forged video detection method based on a weighted feature pyramid provided by the application;

[0023] Figure 2 is a spatial feature extraction network structure diagram of the application;

[0024] Figure 3 is a sensitive point extraction network structure diagram of the application;

[0025] Figure 4 is a fusion forged attention map attention mechanism structure diagram of the application; DETAILED DESCRIPTION

[0026] The application mainly realizes a deep forged video detection method based on a weighted feature pyramid. The specific method adopted by the application will be described in detail below with reference to the drawings.

[0027] Specifically, the process of a deep forged video detection method based on a weighted feature pyramid is as shown in Figure 1 , which includes the following steps: S1: constructing a spatial feature extraction network; S2: constructing a multi-task learning network based on an explicit attention mechanism; S3: training a deep forged video detection model based on a weighted feature pyramid; and S4: inputting the processed video data into the trained deep forged video detection network based on a weighted feature pyramid to determine whether the video is forged.

[0028] For S1: constructing a spatial feature extraction network.

[0029] In the application, the network structure design of the spatial feature extraction network is shown in Figure 2 The spatial feature extraction network is composed of a spatial inconsistency module and a channel attention mechanism module, and mainly includes a convolutional layer 1, a network module 1, a convolutional layer 2, a global average pooling layer, and a convolutional layer 3, wherein each global extraction convolutional layer contains a normalization layer and a nonlinear activation layer.

[0030] EfficientNetV2 is used as the backbone to extract multi-scale features, and its output usually includes multiple down-sampled feature layers (C3, C4, C5), corresponding to down-sampling ratios of 8, 16, and 32. After C5, max pooling is used to generate P6 (stride = 64) and P7 (stride = 128) feature layers. The purpose of this is to better accommodate BiFPN. The spatial resolutions of the five input levels (P3-P7) of BiFPN are p3=48×48; p4=24×24; p5=12×12; p6=6×6, p7=3×3. Each layer first uses a 1×1 convolution to reduce (or increase) the channels to 256 to unify the dimensions.

[0031] A 5-layer BiFPN in series is used to fuse the above multi-scale features. Each BiFPN layer includes bidirectional fusion from top to bottom and from bottom to top: first, the top-down path: starting from the highest layer P7, P7 is up-sampled to 6×6, and P6 is weighted and fused to generate a new P6, and the new P6 is up-sampled to 12×12 and fused with P5, and so on to P3 (48×48). Then the bottom-up path: after completing the bottom-up fusion, starting from the lowest layer P3, the updated P3 is down-sampled and fused with the original P4 to generate a new P4, and then to P7.

[0032] When performing fusion operations, weighted summation (Fast-Normalized Fusion) is used each time, that is, the values of each feature layer are simply added, so the spatial state is not learned. Therefore, a 3×3 depth separable convolution is connected. A 3×3 spatial receptive field operation is performed on each channel to re-aggregate the information of neighboring pixels and enhance the spatial consistency of the fused features. To ensure consistency of the number of channels before and after fusion, a 1×1 convolution is used before and after addition to adjust the channels, which is to maintain consistency of the channels. That is, first do 1×1 before entering, then weighted sum, then 1×1, then 3×3.

[0033] CBAM with fake attention map is introduced for supervision, named FA-CBAM, as Figure 4The spatial attention module of CBAM replaces the original 7x7 convolution with a 3x3 dilated convolution (DC) to enhance the receptive field without changing the parameters. Meanwhile, the supervision function of the multi-task learning branch is combined to transmit the mask of the learnable mask branch in reverse, add an auxiliary loss to the spatial attention output of FA-CBAM, and make it align with the fake mask predicted by the mask branch. The channel number of the intermediate layer of the multi-layer perceptron (MLP) is reduced to 8, which reduces the parameter quantity while still retaining the channel feature information. Then, FA-CBAM is inserted before and after each feature fusion of BiFPN, that is, FA-CBAM is applied to the input and output features of the fusion respectively. The specific method can be: first, each of the two input features to be fused passes through FA-CBAM, then performs weighted fusion convolution, and then adds another FA-CBAM to the fusion output. Use the default channel compression ratio of FA-CBAM, which is 16, and use the structure of ReLU activation and Sigmoid output, without changing the original BiFPN fusion strategy. Finally, a predicted mask is generated .

[0034] For S2: Construct a multi-task learning network based on explicit attention mechanism.

[0035] Generate pseudo-labels on the dataset using Blending-based and adjust the size. Initialize the picture to 0, design the value of the ROI box covered place to 1, and then downsample it to keep the size consistent and pixel-level alignment. Get the label (size 384x384) as the supervision label.

[0036] Design a three-branch structure: classification branch, learnable mask branch, and enhanced self-consistency branch. The classification branch is used to judge the authenticity of the final output result. Because high-quality detection is required, the feature map should be as detailed as possible. Therefore, the multi-level output is subjected to a bilinear interpolation operation and upsampled to the smallest receptive field feature map size. Use the multi-scale output layer of the above BiFPN - Since the resolution of is the largest, select as the feature map benchmark, and adjust the resolution of the other output layers to this benchmark. The specific operation is: bilinearly upsample - to 48x48 in proportion. That is: x2→48; x4→48; x8→48; ×16→48. In this way, each scale of the feature layer can obtain a 48x48x256 feature map. Then all the up-sampled features are spliced in the channel direction to obtain a large feature map of 48x48x1280 (5x256=1280). Then a 1x1 convolution can be used to reduce the channel to 256, and a ReLU activation function is added. Then the global average pooling is performed on the fused features to obtain a 256-dimensional vector, and each feature layer becomes a number, and then the binary (true / false) prediction is output through the fully connected layer. The loss function uses the cross-entropy loss function L0 (Cross-Entropy), and combines the label smoothing regularization (smooths the real label 1 to 0.9) to prevent “too accurate”.

[0037] The learnable mask branch is used as an explicit training strategy, which learns the sensitive point label, and the extraction of the sensitive point label is as shown in Figure 3 . The explicit feature region that the guide network focuses on is as shown in Figure 1 . The input feature layers are and . The unified fusion size is obtained by first up-sampling twice and then splicing in the channel dimension to obtain a feature map F. For the spliced F, a multi-layer convolution and up-sampling are used to recover to the original picture size step by step, because the self-generated label is used in the data set processing stage, in order to change the feature map size to the size of the original picture, the feature learning of the fake region and the explicit guidance.

[0038] The enhanced self-consistency branch is used to guide the learnable mask branch, and is used to enhance the generalization ability, and the structure is as shown in Figure 1 . The perturbation mechanism is added: based on the feature map F of the learnable mask branch, another branch is copied, and Dropout(p=0.2) is applied to it and then LayerNorm is performed, to obtain the perturbation feature F'. The specific method is: randomly discard 20% of the channels on F (the size of which is 24x24x512), and then independently perform Layer Normalization on each sample channel. Consistency prediction: the original feature F and the perturbed F' are sent into the same decoding network to obtain two mask outputs M1 (original) and M2 (perturbed). Since the network structure is the same, only the input is slightly different, the two should have similar predictions. Discrimination loss: introduce L1 loss to constrain the similarity L2 of M1 and M2 in space, and add the total loss with a weight λ2=0.5. This can improve the robustness of the model to feature perturbation and promote consistency learning.

[0039] For S3: a deep fake video detection model based on a weighted feature pyramid is trained.

[0040] The specific steps of training the deep fake video detection model based on the weighted feature pyramid are: using a RetinaNet face extractor to perform face extraction on the video frames, and the resolution of the input image is adjusted to 384x384 after cropping. The backbone network selects EfficientNetV2-S pre-trained on ImageNet, and the number of layers of the backbone network is designed to be 7 layers. After output by F-BiFPN, the 4th and 5th layers are selected as the medium-high layer output; the data set is divided into training set, validation set and test set and is subjected to rotation and scaling processing; the validation set is used to adjust the hyperparameters, and finally the test set is used to test the model effect.

[0041] The steps of dividing the fake video data set into training set, validation set and test set and performing standardization processing are: dividing the data set into training set, validation set and test set in the ratio of 3:1:1.

[0042] In order to expand the data set and prevent overfitting, image enhancement methods such as random rotation and random cropping are used to perform data enhancement on the training set images. Further, when training the network model, a joint loss function is selected, the loss between the model output outputs and the true label labels is calculated, and the training is performed for a total of 200 epochs. After the end of each epoch, 16 frames of images are randomly extracted to generate fake samples, and the first 20% of the epochs are frozen to accelerate the convergence of the downstream task. The optimizer uses SAM, and the initial learning rate is set to 1x10⁻ 4 , which remains unchanged until the end of the first 20% training stage, and then decays to 0 using a weight of 10⁻ 4 . The batch size is 32, and the training data is enhanced by random cropping, scaling, and horizontal flipping, and label smoothing is introduced in the classification branch, which is set to 0.1 for regularization processing. The channel dropout rate p of the fake attention map branch is set to 0.2, and the ε of LayerNorm is set to 1x10⁻ 5 .

[0043] For S4: input the processed video data into the trained deep fake video detection network based on the weighted feature pyramid, and determine whether the video is fake.

[0044] Input the processed video data into the trained deep fake video detection network based on the weighted feature pyramid, and the output of the network model is an area under the curve. By changing the threshold setting, it is determined whether the judgment is accurate, and the final result is displayed as the accuracy of the true or false video judgment.

[0045] The above detailed description is only used to explain the technical solutions of the present application, and is not intended to limit the present application. It should be understood by those skilled in the art that the above embodiments are not in any form to limit the present application, and any similar technical solutions obtained by equivalent replacement or equivalent transformation, etc. belong to the protection scope of the present application.

Claims

1. A method for detecting deepfake videos based on weighted feature pyramids, characterized in that: Includes the following steps, Step 1: Construct a weighted bidirectional feature pyramid network that integrates forged attention maps: EfficientNetV2 is used as the backbone to extract multi-scale features. Its output includes multiple downsampled feature layers C3, C4, and C5, corresponding to downsampling ratios of 8, 16, and 32. Max pooling is used after C5 to generate feature layers P6 with a stride of 64 and P7 with a stride of 128. The five input layers of BiFPN are also described. - The spatial resolution is 3 = 48 × 48; 4 = 24 × 24; 5 = 12 × 12; 6 = 6 × 6; 7 = 3×3, each layer first uses 1×1 convolution to reduce the number of channels to 256 to unify the dimensions; A spoofing attention map based on CBAM is introduced, incorporating dilated convolutions and learnable masks into CBAM. Spatial and channel information are weighted for the spoofing region during each feature fusion. To fully exploit multi-scale information, the spatial attention module of FA-CBAM replaces the original 7×7 convolution with 3×3 dilated convolutions. The supervision function of the multi-task learning branch is combined, and the mask of the learnable mask branch is backpropagated to add an auxiliary loss to the spatial attention output of FA-CBAM, aligning it with the spoofing mask predicted by the mask branch. The number of intermediate channels in the multilayer perceptron (MLP) is reduced to 8, reducing the number of parameters while still retaining channel feature information. Step 2: Construct a multi-task learning network based on explicit attention mechanisms: Pseudo-labels are generated and resized on the dataset using a blending-based approach. The images are initialized to 0, and the values ​​of the areas covered by the ROI boxes are set to 1. Then, downsampling is performed to maintain consistent size and pixel-level alignment. Get tags As a supervisory label; The design employs a three-branch structure: a classification branch, a learnable mask branch, and an enhanced self-consistency branch; the classification branch is used to evaluate the authenticity of the final output result. The multi-level outputs are subjected to bilinear interpolation and upsampled to the minimum receptive field feature map size; a multi-scale output layer using BiFPN is employed. - ; Select As a baseline for the feature maps, other output layers are also adjusted to this baseline resolution; Learnable mask branches can be used as an explicit training strategy, explicitly guiding the network to focus on specific feature regions by learning sensitive point labels; and As an input feature layer; Unify and integrate sizes, first After upsampling by 2 times and The feature map F is obtained by concatenating the channels; for the concatenated F, multi-layer convolution and upsampling are used to restore it to the original image size step by step. The enhanced self-consistency branch is used to guide the learnable mask branch and enhance the generalization ability. Based on the feature map F of the learnable mask branch, another branch is copied, and Dropout is applied to it first, followed by LayerNorm to obtain the perturbation feature F', which is used for consistent supervised learning of F. Step 3: Train and build a deepfake video detection model based on a weighted feature pyramid: The specific steps for training a deepfake video detection model based on a weighted feature pyramid are as follows: The RetinaNet face extractor was used to extract faces from video frames. The input image was cropped and its resolution was adjusted to 384×384. The backbone network uses EfficientNetV2-S pre-trained on ImageNet, with 7 layers. After passing through F-BiFPN, layers 4 and 5 are selected as the mid-to-high-level outputs. The steps for dividing the fake video dataset into training, validation, and test sets and standardizing it are as follows: the dataset is divided into training, validation, and test sets in a 3:1:1 ratio; then rotation and scaling are performed; the validation set is used to adjust hyperparameters; and finally, the model performance is verified using the test set. Step 4: Input the processed video data into the trained weighted feature pyramid deep fake video detection network. The output of the network model is a probability, which shows the accuracy of the final result in judging whether the video is real or fake.

2. The method for detecting deepfake videos using a weighted feature pyramid according to claim 1, characterized in that: The dataset is processed using an explicit supervision mechanism to obtain sensitive point label information, and the ROI is adjusted to maintain consistent size and pixel-level alignment.

3. The method for detecting deepfake videos using a weighted feature pyramid according to claim 1, characterized in that: The RetinaNet sampling point detector is used to extract face data from video frames.

4. The method for detecting deepfake videos using a weighted feature pyramid according to claim 1, characterized in that: When training the network model, a joint loss function was selected to calculate the loss between the model outputs and the true labels. Adam was selected as the optimizer supervised by the binary cross-entropy loss, and the model was trained and validated in multiple rounds.

5. The method for detecting deepfake videos using a weighted feature pyramid according to claim 1, characterized in that: The feature extraction network consists of two parts: a weighted bidirectional feature pyramid network that integrates fake attention maps and a multi-task learning network based on an explicit attention mechanism. It is composed of a BiFPN bidirectional fusion module, upsampling, normalization, nonlinear activation, and attention mechanism modules.

6. The method for detecting deepfake videos using a weighted feature pyramid according to claim 1, characterized in that: The original 7×7 convolution is replaced with a 3×3 dilated convolution to enhance the receptive field without changing the parameters. A fake attention graph module is designed as a feature supervision in the bidirectional fusion process.

7. The method for detecting deepfake videos using a weighted feature pyramid according to claim 1, characterized in that: We design a multi-task learning network that supervises the model by extracting explicit labels from the dataset and uses perturbations to enhance self-consistency.

8. The method for detecting deepfake videos using a weighted feature pyramid according to claim 1, characterized in that: The processed video data is input into the trained weighted feature pyramid deep fake video detection network, and the output is the area under the curve. The accuracy of the judgment is determined by changing the set threshold. When the area is higher than 0.5, the video is judged accurately, that is, real video is judged as real and fake video is judged as fake; otherwise, the judgment is inaccurate.

Images