Face change video detection method and system based on illumination feature decoupling

Abstract

The invention discloses a face-changing video detection method and system based on illumination feature decoupling. The method comprises the following steps of: framing a video and extracting an RGB (Red, Green and Blue) image of a detection area; sending the RGB image into a learnable local gravity mode extraction module, and extracting a local gravity mode image; the RGB image and the local gravity mode image are sent to a double-branch high-semantic feature extraction module, and illumination related features and illumination invariant features are extracted respectively; the illumination related features are sent to a contrast learning module based on relighting; sending the classification features into a classifier to obtain prediction probability distribution, and carrying out dichotomy supervision by using cross entropy loss; training a model and storing the model; and the model test loading model outputs a detection result of the to-be-detected video. According to the method, the illumination features are decoupled by using the domain generalization strategy, interference of domain-related illumination information is effectively suppressed, the characterization capability of the detection features is improved, and the method has a relatively good detection effect and generalization capability.

Description

Technical Field

[0001] The present invention relates to the technical field of digital video tampering detection, and in particular to a face-swap video detection method and system based on illumination feature decoupling. Background Art

[0002] With the advancement of artificial intelligence (AI) technology driven by generative models, facial identity replacement video synthesis technology, centered around deepfakes, has garnered widespread social attention. While enabling innovation in the digital content industry (such as film and television production, immersive virtual experiences, and interactive entertainment), this technology also poses a substantial threat to national information security, social order, and citizens' privacy and property. Therefore, researching reliable defenses against face-swapping videos has become a pressing need, possessing significant strategic value and practical significance.

[0003] Currently, the primary approach and research focus for defending against face-swapped videos lies in passive detection techniques. The core issue in passive detection research is improving the generalization capabilities of detection methods (i.e., their ability to detect unknown face-swapped video data) to facilitate their application in real-world scenarios. Existing approaches to improving model generalization in passive detection techniques can be categorized into two main categories: enhancing the detection model's feature representation capabilities and expanding the detection model's training environment. The former improves feature representation capabilities by introducing artificially designed prior knowledge or combining multimodal features, while the latter expands the training environment by generating fake face-swapped samples or performing model transfer (including task transfer and data domain transfer). Both approaches can effectively improve the generalization capabilities of detection models. However, current research on the generalization performance of passive detection models still faces significant challenges. First, the continuous iteration and upgrading of face-swapped forgery algorithms places increasing pressure on passive detection methods. Second, the complex characteristics of real-world face-swapped video data (such as diverse shooting environments and diverse post-processing operations) make it difficult for existing methods to maintain good detection performance in open environments. In this context, the study of the generalization performance of passive face-changing video detection technology remains a key scientific issue that needs to be solved urgently. Summary of the Invention

[0004] In order to overcome the defects and shortcomings of the existing technology, the present invention provides a face-swap video detection method and system based on illumination feature decoupling. The present invention aims to explore more essential tampering traces by decoupling illumination features from the perspective of the image imaging environment. First, a learnable local gravity pattern image (LPLGF) feature extraction module is used to extract the local gravity pattern image from the RGB image. Then, a dual-branch high-semantic feature extraction module is used to extract illumination-related features and illumination-invariant features with high-level semantics from the RGB image and the local gravity pattern image, respectively. A contrastive learning module based on re-illumination is used to enhance the authenticity-related illumination information in the illumination-related features and suppress the illumination information related to the data domain. An adversarial learning module is used to enhance the domain-invariant properties of the illumination-invariant features. Finally, a hierarchical feature fusion module is used to fuse the features of different levels in the dual-branch feature extraction network to obtain high-semantic classification features. The detection features extracted in this way have good versatility and can effectively improve the generalization performance of the detection algorithm.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] Decode the video into a frame sequence and extract the RGB image I of the detection area of ​​each frame frame by frame RGB ;

[0007] Will I RGB The LPLGF feature extraction module is used to extract the local gravity pattern image I. LPLGF ;

[0008] Will I RGB and I LPLGF Send it to the dual-stream high semantic feature extraction module to extract the illumination-related features f R,k and illumination-invariant features f I,k , k represents the level of the feature extraction network used, k = [1,4];

[0009] The illumination-related features f output by the last layer of the dual-stream high semantic feature extraction module are R,4 Send it to the Relighting-based Contrastive Learning Module (RCLM) and use the contrast loss L ctr Enhance the illumination information related to authenticity and suppress the illumination information related to the data domain;

[0010] The illumination invariant feature f output by the last layer of the dual-branch high semantic feature extraction module is I,4 Send it to the Adversarial Learning Module (ALM) and use the adversarial loss L adv Enhance the domain-invariant properties of features;

[0011] The Hierarchical Feature Fusion Module (HFFM) is used to fuse the features of different levels in the dual-branch feature extraction network to obtain the highly semantic classification feature f A ;

[0012] The classification feature f A Enter the classifier to get the predicted probability distribution, and use the cross entropy loss L cls Conduct binary supervision;

[0013] Model training, updating network parameters with the goal of minimizing the total loss function, and saving the network model and parameters after training is completed;

[0014] Model testing: load the trained detection model and output the detection results of the video to be tested.

[0015] As a preferred technical solution, the video is decoded into a frame sequence, and the RGB image I of each frame detection area is extracted frame by frame. RGB , the specific steps include:

[0016] The video of the multi-domain training dataset is divided into frames, and Dlib is used to detect the 68 facial key points of the face in each frame. The circumscribed rectangle of the 68 facial key points is used as the position of the face rectangle frame, and 1.3 times the face is selected as the detection area to obtain the RGB image of the detection area of ​​each frame. Among them, 1, 3, 256, and 256 are the number of image samples, the number of channels after image normalization, and the height and width respectively.

[0017] As a preferred technical solution, the RGB The LPLGF feature extraction module is used to extract the local gravity pattern image I. LPLGF , the specific steps include:

[0018] Will I RGB is fed into the LPLGF feature extraction module, which first uses a learnable local gravity pattern operator w LPLGF In the horizontal and vertical directions, I RGB Perform convolution to obtain the horizontal gradient IGH and vertical gradient I GV ; Then use the Lambert model to calculate I GH and I GV Perform light separation to obtain the horizontal light separation gradient I SGH and vertical illumination separation gradient I SGV ; Finally, use I SGH and I SGV Calculate the local gravity pattern image I LPLGF .

[0019] The learnable local gravity mode operator w LPLGF First, initialize two 3×3 convolution kernels w H and w V , then use w H and w V Calculate the horizontal gradient I separately GH and vertical gradient I GV , where w H and w V The convolution kernel parameters can be automatically updated according to the training data, which has better adaptive feature extraction capabilities.

[0020] The w H and w V The initialization kernel parameters are as follows:

[0021]

[0022] The horizontal gradient I GH and vertical gradient I GV The calculation formula is as follows:

[0023] I GH =I RGB *w H , I GV =I RGB *w V

[0024] Where * represents a two-dimensional convolution operation.

[0025] The light separation SGH and I SGV The calculation formula is as follows:

[0026]

[0027] Here, λ is a stable constant whose value is a very small positive number to prevent numerical anomalies caused by the denominator being 0.

[0028] The local gravity pattern image I LPLGF The calculation formula is as follows:

[0029]

[0030] Where arctan() represents the inverse tangent activation function, which is used to provide nonlinear transformation so that the neural network can learn more complex feature patterns.

[0031] As a preferred technical solution, the RGB and I LPLGF Send it to the dual-stream high semantic feature extraction module to extract the illumination-related features f R,k and illumination-invariant features f I,k , k represents the level of the feature extraction network used, k = [1, 4], and the specific steps include:

[0032] First of all, I RGB and I LPLGF Perform difference operation and absolute value operation to obtain the illumination-related difference image I Diff , then I RGB and I Diff After splicing along the channel direction, it is sent to the first branch of the dual-branch high semantic feature extraction module, and I LPLGF The second branch of the dual-stream high semantic feature extraction module extracts the illumination-related features f R,k and illumination-invariant features f I,k , where k represents the level of the feature extraction network used, k = [1,4].

[0033] As a preferred technical solution, the illumination-related feature f outputted by the last layer of the dual-stream high semantic feature extraction module is extracted. R,4 Send it to the Relighting-based Contrastive Learning Module (RCLM) and use the contrast loss L ctr Enhance the authenticity-related illumination information and suppress the data domain-related illumination information. The specific steps include:

[0034] The illumination-related features f output by the last layer of the dual-stream high semantic feature extraction module are R,4 Send it to RCLM for feature optimization, using contrast loss L ctr Enhance the authenticity-related illumination information and suppress the data domain-related illumination information.

[0035] The specific steps of the RCLM include: first, training batch data I from a multi-domain dataset i Shuffle to get data after the sample order is disrupted where i * =random{1,2,…,N}, N represents the number of samples in the current training batch data; then Ii and Input the relighting model F RL , using F RL Will The lighting information is passed to I in a sample-by-sample manner i , to generate training samples with randomized illumination Finally, the same Send it to the dual-stream high semantic feature extraction module to obtain its corresponding illumination-related features And use contrast loss L ctr f R,4 and Perform feature optimization to enhance the authenticity-related illumination information and suppress the data domain-related illumination information.

[0036] The heavy lighting model F RL Use the existing open source re-lighting model implementation, whose model weights are frozen and not involved in training, only used for generation

[0037] The contrast loss L ctr The calculation formula is as follows:

[0038]

[0039] Where E(·) is used to measure the label consistency of the feature, label(·) represents the true and false labels corresponding to the sample features, Sim(·) represents the negative cosine similarity, and ||·||2 represents the L2 norm.

[0040] As a preferred technical solution, the illumination invariant feature f output by the last layer of the dual-stream high semantic feature extraction module is I,4 Send it to the Adversarial Learning Module (ALM) and use the adversarial loss L adv Enhance the domain invariance of features. The specific steps include:

[0041] The illumination invariant feature f output by the last layer of the dual-branch high semantic feature extraction module is I,4 Send it to ALM for feature optimization and use adversarial loss L adv Enhance the domain-invariant properties of features.

[0042] The adversarial loss L adv The calculation formula is as follows:

[0043]

[0044] where X and Y D Represent the sample set and domain label set respectively, N Dis the number of data domains. 1[i=y] means that when the domain number i is equal to the domain type prediction result y, the output value is 1, otherwise it is 0. G represents the generation of f I,4 The feature extraction network is denoted by G, and D represents the domain discriminator. In order to optimize both G and D simultaneously, this method uses the Gradient Reversal Layer (GRL) to multiply the gradient by a negative scalar during backpropagation to achieve gradient reversal.

[0045] As a preferred technical solution, the hierarchical feature fusion module (HFFM) is used to fuse the features of different levels in the dual-branch feature extraction network to obtain a high-semantic classification feature f A , the specific steps include:

[0046] HFFM is used to fuse the features of different levels in the dual-branch feature extraction network to obtain the highly semantic classification feature f A .

[0047] The f A The calculation formula is as follows:

[0048]

[0049] Where k represents the network level of the dual-stream high semantic feature extraction module, k = [1, 4], f A,k represents the features after the k-th layer network fusion, f A,0 =0. represents the HFFM operation of the kth layer, Represents a two-dimensional convolution operation, which is used to align the channel dimensions of features at different levels. Represents an upsampling operation, which is used to align the spatial dimensions of features at different levels for element-wise addition.

[0050] As a preferred technical solution, the classification feature f A Enter the classifier to get the predicted probability distribution, and use the cross entropy loss L cls Perform binary classification supervision. The specific steps include:

[0051] The classification feature f A Enter the classifier to get the predicted probability distribution p, and use the cross entropy loss function L cls To conduct supervision;

[0052] The cross entropy loss L cls The calculation formula is as follows:

[0053]

[0054] where NC Indicates the number of categories, that is, N C =2, p and y represent the predicted probability and classification label respectively.

[0055] As a preferred technical solution, the model training updates the network parameters with the goal of minimizing the total loss function, and saves the network model and parameters after the training is completed. The specific steps are as follows:

[0056] Model training updates the network parameters with the goal of minimizing the total loss function, and saves the network model and parameters after training is completed.

[0057] The total loss function L is as follows:

[0058] L=L cls +αL map

[0059] Where α is a hyperparameter that balances the two loss functions.

[0060] As a preferred technical solution, the model training updates the network parameters with the goal of minimizing the total loss function, and saves the network model and parameters after the training is completed. The specific steps are as follows:

[0061] The total loss function L is as follows:

[0062] L=L cls +α·K ctr +β·L adv

[0063] Among them, α and β are hyperparameters that balance the two loss functions.

[0064] As a preferred technical solution, the model test loads a trained detection model and outputs the detection result of the video to be tested.

[0065] The present invention also provides a face-swap video detection system based on illumination feature decoupling, comprising: a data preprocessing module, a learnable local gravity pattern feature extraction module, a dual-branch high semantic feature extraction module, a contrast learning module based on heavy illumination, an adversarial learning module, a hierarchical feature fusion module, a binary classification supervision module, a training module, and a testing module;

[0066] The data preprocessing module is used to decode the video into a frame sequence and extract the RGB image of each frame detection area frame by frame. RGB ;

[0067] The learnable local gravity pattern feature extraction module is used to extract the I RGBThe LPLGF feature extraction module is used to extract the local gravity pattern image I. LPLGF ;

[0068] The dual-branch high semantic feature extraction module is used to RGB and I LPLGF Send it to the dual-stream high semantic feature extraction module to extract the illumination-related features f R,k and illumination-invariant features f I,k , k represents the level of the feature extraction network used, k = [1,4];

[0069] The contrastive learning module based on re-illumination is used to transform the illumination-related features f output by the last layer of the dual-branch high semantic feature extraction module into R,4 Send it to the Relighting-based Contrastive Learning Module (RCLM) and use the contrast loss L ctr Enhance the illumination information related to authenticity and suppress the illumination information related to the data domain;

[0070] The adversarial learning module is used to transform the illumination invariant feature f output by the last layer of the dual-branch high semantic feature extraction module I,4 Send it to the Adversarial Learning Module (ALM) and use the adversarial loss L adv Enhance the domain-invariant properties of features;

[0071] The hierarchical feature fusion module is used to fuse the features of different levels in the dual-branch feature extraction network using the hierarchical feature fusion module (HFFM) to obtain a high-semantic classification feature f A ;

[0072] The two-class supervision module is used to classify the classification feature f A Enter the classifier to get the predicted probability distribution, and use the cross entropy loss L cls Conduct binary supervision;

[0073] The training module is used for model training, updates network parameters with the goal of minimizing the total loss function, and saves the network model and parameters after training is completed;

[0074] The test module is used for model testing, loading the trained detection model, and outputting the detection results of the video to be tested.

[0075] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0076] (1) The present invention designs a domain generalization framework based on illumination feature decoupling, extracting more essential detection features from the perspective of the image imaging environment. By decoupling more general illumination features, the detection model’s dependence on the illumination of a specific data domain environment is alleviated, thereby improving the generalization ability of the detection model in data domains with unknown illumination environments.

[0077] (2) The present invention designs a learnable local gravity pattern extraction module to extract the local gravity pattern image, and sends it and the RGB image to the dual-branch high semantic feature extraction module to extract illumination-related features and illumination-invariant features. Finally, the hierarchical feature fusion module is used to fuse the two-branch features, effectively improving the representation ability of the detection features.

[0078] (3) The present invention designs a contrast learning module based on heavy illumination to decouple illumination features, uses contrast loss to enhance the illumination information related to authenticity in illumination-related features, and suppresses illumination information related to the data domain, effectively improving the authenticity identification ability and cross-domain robustness of illumination features.

[0079] (4) The present invention designs an adversarial learning module to perform feature domain alignment, and uses adversarial loss to enhance the domain-invariant properties of illumination-invariant features, effectively improving the generalization ability of detection features in unseen data domains. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 This is a schematic flow chart of the face-swap video detection method based on illumination feature decoupling in this embodiment; (Note: The facial image has been blurred to ensure facial security)

[0081] Figure 2 Schematic diagram of real and forged face images and their corresponding local gravity pattern images in the face-swap video detection method based on illumination feature decoupling in this embodiment; (Note: The face images have been blurred to ensure face security)

[0082] Figure 3 This is a schematic diagram of the effect of generating a re-illumination model in the face-swap video detection method based on illumination feature decoupling in this embodiment; (Note: The facial image has been blurred to ensure facial security)

[0083] Figure 4 This is a schematic diagram of the process steps of the model training part in the face-swap video detection method based on illumination feature decoupling in this embodiment.

[0084] Figure 5 This is a schematic diagram of the process steps of the model testing part of the face-changing video detection method based on illumination feature decoupling in this embodiment. DETAILED DESCRIPTION

[0085] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0086] Example

[0087] This example uses four representative face forgery video databases: FaceForensics++ (FF++), DeepfakeDetection (DFD), DeepfakeDetection Challenge (DFDC), and Celeb-DF for training, validation, and testing. The FF++ database is a benchmark for face forgery detection. It consists of 1,000 real face videos collected from YouTube. Each real video is generated using four face manipulation techniques: DeepFake (DF), Face2Face (F2F), FaceSwap (FS), and NeuralTextures (NT). The output videos are compressed at different rates: original quality (c0), high quality (c23), and low quality (c40) to simulate real-world video compression. The DFD database consists of 363 real videos and 3,068 forged videos shot by 28 actors. It also offers three compression rates: c0, c23, and c40. These videos encompass a variety of life scenarios and facial expressions, resulting in realistic forgeries. The DFDC database consists of 23,654 real videos and 104,500 fake videos shot by 3,426 paid actors. The fakes are highly realistic, making it a representative and challenging face fakery database. The Celeb-DF database includes 590 real face videos from YouTube and 5,639 high-quality fake face videos, each with a very realistic fake quality.

[0088] This example uses the FF++(c23) database for training and validation, the FF++(c23) database for intra-database testing, and the DFD(c23), DFDC, and Celeb-DF databases for cross-database testing. The implementation of this example is detailed. The database is divided into training, validation, and test sets in a ratio of 7:2:1. This example is simulated on a Linux system (Ubuntu 18.04) using Python 3.7 and the deep learning framework Pytorch 1.8.0. The graphics card used is an RTX 3090 with CUDA version 11.4.

[0089] like Figure 1As shown, this embodiment provides a face-swap video detection method based on illumination feature decoupling, comprising the following steps:

[0090] S1: Decode the video into a frame sequence and extract the RGB image I of the detection area of ​​each frame frame by frame RGB ;

[0091] In this example, we first use OpenCV to divide the video of the multi-domain training dataset into frames, use Dlib to detect the 68 facial key points of each frame, and use the circumscribed rectangle of the 68 facial key points as the position of the face rectangle. We select 1.3 times the face as the detection area and obtain the RGB image of the detection area of ​​each frame. Among them, 1, 3, 256 and 256 are the number of image samples and the number of channels, height and width after image normalization respectively.

[0092] S2: I RGB The LPLGF feature extraction module is used to extract the local gravity pattern image I. LPLGF ;

[0093] In this embodiment, I RGB is fed into the LPLGF feature extraction module, which first uses a learnable local gravity pattern operator w LPLGF In the horizontal and vertical directions, I RGB Perform convolution to obtain the horizontal gradient I GH and vertical gradient I GV ; Then use the Lambert model to calculate I GH and I GV Perform light separation to obtain the horizontal light separation gradient I SGH and vertical illumination separation gradient I SGV ; Finally, use I SGH and I SGV Calculate the local gravity pattern image I LPLGF . Figure 2 Schematic diagram of real face images, fake face images and their corresponding local gravity pattern images.

[0094] The learnable local gravity mode operator w LPLGF First, initialize two 3×3 convolution kernels w H and w V , then use w H and w V Used to calculate the horizontal gradient I GH and vertical gradient I GV , where w Hand w V The convolution kernel parameters can be automatically updated according to the training data, which has better adaptive feature extraction capabilities.

[0095] The w H and w V The initialization kernel parameters are as follows:

[0096]

[0097] The horizontal gradient I GH and vertical gradient I GV The calculation formula is as follows:

[0098] I GH =I RGB *w H , I GV =I RGB *w V

[0099] Where * represents a two-dimensional convolution operation.

[0100] The light separation SGH and I SGV The calculation formula is as follows:

[0101]

[0102] Here, λ is a stable constant whose value is a very small positive number (λ=0.001 in this embodiment) to prevent numerical anomalies caused by the denominator being 0.

[0103] The local gravity pattern image I LPLGF The calculation formula is as follows:

[0104]

[0105] Where arctan() represents the inverse tangent activation function, which is used to provide nonlinear transformation so that the neural network can learn more complex feature patterns.

[0106] S3: I RGB and I LPLGF Send it to the dual-stream high semantic feature extraction module to extract the illumination-related features f R,k and illumination-invariant features f I,k , k represents the level of the feature extraction network used, k = [1,4];

[0107] In this embodiment, first, I RGB and I LPLGF Perform difference operation and absolute value operation to obtain the illumination-related difference image I Diff , then I RGBand I Diff After splicing along the channel direction, it is sent to the first branch of the dual-branch high semantic feature extraction module, and I LPLGF The second branch of the dual-stream high semantic feature extraction module extracts the illumination-related features f R,k and illumination-invariant features f I,k , where k represents the level of the feature extraction network used, k = [1,4].

[0108] In this embodiment, each tributary network in the dual-tributary high semantic feature extraction module is implemented using the ResNet-18 network architecture.

[0109] S4: The illumination-related features f output by the last layer of the dual-stream high semantic feature extraction module are extracted R,4 Send it to the Relighting-based Contrastive Learning Module (RCLM) and use the contrast loss L ctr Enhance the illumination information related to authenticity and suppress the illumination information related to the data domain;

[0110] In this embodiment, the illumination-related features f output by the last layer of the dual-stream high semantic feature extraction module are R,4 Send it to RCLM for feature optimization, using contrast loss L ctr Enhance the authenticity-related illumination information and suppress the data domain-related illumination information.

[0111] The specific steps of the RCLM include: first, training batch data I from a multi-domain dataset i Shuffle to get data after the sample order is disrupted where i * =random{1,2,…,N}, N represents the number of samples in the current training batch data; then I i and Input the relighting model F RL , using F RL Will The lighting information is passed to I in a sample-by-sample manner i , to generate training samples with randomized illumination Finally, the same Send it to the dual-stream high semantic feature extraction module to obtain its corresponding illumination-related features And use contrast loss L ctr f R,4 and Perform feature optimization to enhance the authenticity-related illumination information and suppress the data domain-related illumination information.

[0112] The heavy lighting model F RL Use the existing open source re-illumination model implementation (this embodiment uses the open source face re-illumination model GeomConsistentFR, published in the top international conference on computer vision and pattern recognition in 2022). Its model weights are frozen and do not participate in training. They are only used for generating Figure 3 This is a schematic diagram of the effect of the re-lighting model generation. From the results, we can see that the re-lighting model can well transfer the lighting information of the reference image to the original image. The generated image has both the image content of the original image and the lighting information of the reference image, and the generation quality is good.

[0113] The contrast loss L ctr The calculation formula is as follows:

[0114]

[0115] Where E(·) is used to measure the label consistency of the feature, label(·) represents the true and false labels corresponding to the sample features, Sim(·) represents the negative cosine similarity, and ||·||2 represents the L2 norm.

[0116] S5: The illumination invariant feature f output by the last layer of the dual-branch high semantic feature extraction module I,4 Send it to the Adversarial Learning Module (ALM) and use the adversarial loss L adv Enhance the domain-invariant properties of features;

[0117] In this embodiment, the illumination invariant feature f output by the last layer of the dual-stream high semantic feature extraction module is converted into I,4 Send it to ALM for feature optimization and use adversarial loss L adv Enhance the domain-invariant properties of features.

[0118] The adversarial loss L adv The calculation formula is as follows:

[0119]

[0120] where X and Y D Represent the sample set and domain label set respectively, N D is the number of data domains. 1[i=y] means that when the domain number i is equal to the domain type prediction result y, the output value is 1, otherwise it is 0. G represents the generation of f I,4The feature extraction network is denoted by

[15] , where D represents the domain discriminator (consisting of three fully connected layers with 512, 256, and 2 neurons, respectively). To optimize both G and D simultaneously, this method uses a gradient reversal layer (GRL) to multiply the gradient by a negative scalar during backpropagation.

[0121] S6: Use the Hierarchical Feature Fusion Module (HFFM) to fuse the features of different levels in the dual-branch feature extraction network to obtain the highly semantic classification feature f A ;

[0122] In this embodiment, HFFM is used to fuse the features of different levels in the dual-branch feature extraction network to obtain the high-semantic classification feature f A .

[0123] The f A The calculation formula is as follows:

[0124]

[0125] Where k represents the network level of the dual-stream high semantic feature extraction module, k = [1, 4], f A,k represents the features after the k-th layer network fusion, f A,0 =0. represents the HFFM operation of the kth layer, Represents a two-dimensional convolution operation, which is used to align the channel dimensions of features at different levels. Represents an upsampling operation, which is used to align the spatial dimensions of features at different levels for element-wise addition.

[0126] S7: Classification feature f A Enter the classifier to get the predicted probability distribution, and use the cross entropy loss L cls Conduct binary supervision;

[0127] In this embodiment, the classification feature f A The classifier (including 3 fully connected layers with 512, 256 and 2 neurons) is used to obtain the predicted probability distribution p and the cross entropy loss function L is used. cls To conduct supervision;

[0128] The cross entropy loss L cls The calculation formula is as follows:

[0129]

[0130] where N C Indicates the number of categories, that is, NC =2, p and y represent the predicted probability and classification label respectively.

[0131] S8: Model training, updating network parameters with the goal of minimizing the total loss function, and saving the network model and parameters after training is completed.

[0132] like Figure 4 As shown, in this embodiment, the training samples are sequentially input into the data preprocessing module, the learnable local gravity pattern feature extraction module, the dual-branch high semantic feature extraction module, the contrastive learning module based on re-illumination, the adversarial learning module, the hierarchical feature fusion module, and the binary classification supervision module, and the network parameters are updated with the goal of minimizing the total loss function. The total loss function L is shown below, where α and β are hyperparameters that balance the two loss functions. In this embodiment, α and β are 1.0 and 1.0, respectively.

[0133] L=L cls +α·L ctr +β·L adv

[0134] This example uses the Adam algorithm for parameter optimization. During the training process, the BatchSize is 32 and the initial learning rate lr is 2×10 -3 The total number of training epochs is set to 30, and the lr is reduced by half every 10 epochs. The decision threshold of the algorithm is fixed at 0.5. Then, the network parameters are updated with the goal of minimizing the total loss function. After training, the network model and parameters are saved.

[0135] S9: Model testing, loading the trained detection model and outputting the detection results of the video to be tested;

[0136] like Figure 5 As shown, in this embodiment, the trained detection model is loaded, the predicted probability distribution is output, and the final prediction result is obtained based on the decision threshold (set to 0.5). The benchmark index of the detection result is calculated based on the predicted result and the actual label value.

[0137] In this embodiment, the performance evaluation metric for the face forgery detection algorithm is the area under the receiver operating characteristic curve (ROC). As shown in Table 1 below, a confusion matrix is ​​used to explain the above metric in detail:

[0138] Table 1 Confusion matrix

[0139] Tags / Predictions Prediction is true Prediction is false Label is true TP FN Label is false FP TN

[0140] The false positive rate (FPR) refers to the ratio of the number of forged faces judged as real faces to the number of faces labeled as forged, that is, the missed detection rate mentioned above; the false negative rate (FNR) refers to the ratio of the number of real faces judged as forged faces to the number of faces labeled as real, that is, the false alarm rate mentioned above.

[0141]

[0142] The area under the receiver operating characteristic curve (AUC) is the area under the receiver operating characteristic (ROC) curve, which is plotted against the FPR at different thresholds and 1-FNR at different thresholds. The AUC is independent of the decision threshold and objectively reflects the overall performance of a classifier at different detection thresholds. A larger AUC indicates a better classifier.

[0143] To demonstrate the effectiveness of the present invention and verify its generalization performance, this example follows the test conditions of existing detection algorithms, performing training and in-database testing on the FF++(c23) database, and cross-database testing on the DFD(c23) database, the DFDC database, and the Celeb-DF database. Furthermore, the benchmark algorithm for face forgery detection, Xception (published in 2017 at the top international conference on computer vision and pattern recognition), and the representative, top-tier MFLE detection algorithm (published in 2024 at a top international journal in computer graphics and multimedia) were selected for comparison. The results of the in-database and cross-database experiments are shown in Table 2:

[0144] Table 2 AUC scores (%) for cross-library detection based on FF++(c23) database training

[0145]

[0146] As shown in Table 2, compared to the Xception method, the proposed method achieves improvements of 2.95%, 10.06%, 12.16%, and 21.22% in AUC scores on the FF++(c23), DFD(c23), DFDC, and CDF datasets, respectively. Compared to the MFLE method, the proposed method achieves improvements of 1.62%, 1.54%, 1.56%, and 2.42% in AUC scores on the FF++(c23), DFD(c23), DFDC, and CDF datasets, respectively. While maintaining high accuracy within the database, the proposed method effectively improves the generalization performance of the detection model, validating its effectiveness and advancement.

[0147] The present invention addresses the problem that existing face-swapped video detection technologies have difficulty extracting universal detection features. It aims to explore more essential traces of tampering by decoupling illumination features from the perspective of the image imaging environment, providing an effective approach for face-swapped video detection. Furthermore, by constructing a learnable local gravity pattern feature extraction module, a dual-stream high-semantic feature extraction module, a contrast learning module based on heavy illumination, an adversarial learning module, and a hierarchical feature fusion module, the characterization capability of detection features is improved, more universal detection features are extracted, and the generalization performance of the detection algorithm is enhanced. The above experimental results demonstrate that this embodiment has excellent cross-database detection performance on existing public databases, fully verifying the effectiveness of the method of the present invention.

[0148] This embodiment also provides a face-swap video detection system based on illumination feature decoupling, comprising: a data preprocessing module, a learnable local gravity pattern feature extraction module, a dual-stream high semantic feature extraction module, a contrastive learning module based on heavy illumination, an adversarial learning module, a hierarchical feature fusion module, a binary classification supervision module, a training module, and a testing module;

[0149] In this embodiment, the data preprocessing module is used to decode the video into a frame sequence, and extract the RGB image I of each detection area frame by frame. RGB ; The learnable local gravity pattern feature extraction module is used to extract I RGB The LPLGF feature extraction module is used to extract the local gravity pattern image I. LPLGF ; The dual-stream high semantic feature extraction module is used to RGB and I LPLGF Send it to the dual-stream high semantic feature extraction module to extract the illumination-related features f R,k and illumination-invariant features f I,k , k represents the level of the feature extraction network used, k = [1, 4]; the contrastive learning module based on heavy illumination is used to transform the illumination-related features f output by the last layer of the dual-branch high semantic feature extraction module R,4 Send it to the Relighting-based Contrastive Learning Module (RCLM) and use the contrast loss L ctr Enhance the illumination information related to authenticity and suppress the illumination information related to the data domain; the adversarial learning module is used to transform the illumination invariant feature f output by the last layer of the dual-branch high semantic feature extraction module I,4 Send it to the Adversarial Learning Module (ALM) and use the adversarial loss L advEnhance the domain invariance of features; the hierarchical feature fusion module is used to fuse features of different levels in the dual-branch feature extraction network using the hierarchical feature fusion module (HFFM) to obtain high-semantic classification features f A ; The binary supervision module is used to classify the classification features f A Enter the classifier to get the predicted probability distribution, and use the cross entropy loss L cls Perform binary classification supervision; the training module is used for model training, updating the network parameters with the goal of minimizing the total loss function, and saving the network model and parameters after training is completed; the testing module is used for model testing, loading the trained detection model, and outputting the detection results of the video to be tested.

[0150] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A face-swap video detection method based on illumination feature decoupling, characterized in that: The steps include: Decode the video into a frame sequence and extract the RGB image I of the detection area of ​​each frame frame by frame RGB ; Will I RGB The LPLGF feature extraction module is used to extract the local gravity pattern image I. LPLGF ; Will I RGB and I LPLGF Send it to the dual-stream high semantic feature extraction module to extract the illumination-related features f R,k and illumination-invariant features f I,k , k represents the level of the feature extraction network used, k = [1,4]; The illumination-related features f output by the last layer of the dual-stream high semantic feature extraction module are R,4 Send it to the Relighting-based Contrastive Learning Module (RCLM) and use the contrast loss L ctr Enhance the illumination information related to authenticity and suppress the illumination information related to the data domain; The illumination invariant feature f output by the last layer of the dual-branch high semantic feature extraction module is I,4 Send it to the Adversarial Learning Module (ALM) and use the adversarial loss L adv Enhance the domain-invariant properties of features; The Hierarchical Feature Fusion Module (HFFM) is used to fuse the features of different levels in the dual-branch feature extraction network to obtain the highly semantic classification feature f A ; The classification feature f A Enter the classifier to get the predicted probability distribution, and use the cross entropy loss L cls Conduct binary supervision; Model training, updating network parameters with the goal of minimizing the total loss function, and saving the network model and parameters after training is completed; Model testing: load the trained detection model and output the detection results of the video to be tested.

2. The face-swap video detection method based on illumination feature decoupling according to claim 1 is characterized in that: The video is decoded into a frame sequence, and the RGB image of the detection area of ​​each frame is extracted frame by frame. RGB , the specific steps include: The video of the multi-domain training dataset is divided into frames, and Dlib is used to detect the 68 facial key points of the face in each frame. The circumscribed rectangle of the 68 facial key points is used as the position of the face rectangle frame, and 1.3 times the face is selected as the detection area to obtain the RGB image of the detection area of ​​each frame. Among them, 1, 3, 256, and 256 are the number of image samples, the number of channels after image normalization, and the height and width respectively.

3. The face-swap video detection method based on illumination feature decoupling according to claim 1 is characterized in that: The I RGB The LPLGF feature extraction module is used to extract the local gravity pattern image I. LPLGF , the specific steps include: Will I RGB is fed into the LPLGF feature extraction module, which first uses a learnable local gravity pattern operator w LPLGF In the horizontal and vertical directions, I RGB Perform convolution to obtain the horizontal gradient I GH and vertical gradient I GV ; Then use the Lambert model to calculate I GH and I GV Perform light separation to obtain the horizontal light separation gradient I SGH and vertical illumination separation gradient I SGV ; Finally, use I SGH and I SGV Calculate the local gravity pattern image I LPLGF . The learnable local gravity mode operator w LPLGF First, initialize two 3×3 convolution kernels w H and w V , then use w H and w V Used to calculate the horizontal gradient I GH and vertical gradient I GV , where w H and w V The convolution kernel parameters can be automatically updated according to the training data, which has better adaptive feature extraction capabilities. The w H and w V The initialization kernel parameters are as follows: The horizontal gradient I GH and vertical gradient I GV The calculation formula is as follows: I GH =I RGB *w H ,I GV =I RGB *w V Where * represents a two-dimensional convolution operation. The light separation SGH and I SGV The calculation formula is as follows: Here, λ is a stable constant whose value is a very small positive number to prevent numerical anomalies caused by the denominator being 0. The local gravity pattern image I LPLGF The calculation formula is as follows: Where arctan() represents the inverse tangent activation function, which is used to provide nonlinear transformation so that the neural network can learn more complex feature patterns.

4. The face-swap video detection method based on illumination feature decoupling according to claim 1 is characterized in that: The I RGB and I LPLGF Send it to the dual-stream high semantic feature extraction module to extract the illumination-related features f R,k and illumination-invariant features f I,k , k represents the level of the feature extraction network used, k = [1, 4], and the specific steps include: First of all, I RGB and I LPLGF Perform difference operation and absolute value operation to obtain the illumination-related difference image I Diff , then I RGB and I Diff After splicing along the channel direction, it is sent to the first branch of the dual-branch high semantic feature extraction module, and I LPLGF The second branch of the dual-stream high semantic feature extraction module extracts the illumination-related features f R,k and illumination-invariant features f I,k , where k represents the level of the feature extraction network used, k = [1,4].

5. The face-swap video detection method based on illumination feature decoupling according to claim 1 is characterized in that: The illumination-related features f output by the last layer of the dual-stream high semantic feature extraction module are R,4 Send it to the Relighting-based Contrastive Learning Module (RCLM) and use the contrast loss L ctr Enhance the authenticity-related illumination information and suppress the data domain-related illumination information. The specific steps include: The illumination-related features f output by the last layer of the dual-stream high semantic feature extraction module are R,4 Send it to RCLM for feature optimization, using contrast loss L ctr Enhance the authenticity-related illumination information and suppress the data domain-related illumination information. The specific steps of the RCLM include: first, training batch data I from a multi-domain dataset i Shuffle to get data after the sample order is disrupted where i * =random{1,2,…,N}, N represents the number of samples in the current training batch data; then I i and Input the relighting model F RL , using F RL Will The lighting information is passed to I in a sample-by-sample manner i , to generate training samples with randomized illumination Finally, the same Send it to the dual-stream high semantic feature extraction module to obtain its corresponding illumination-related features And use contrast loss L ctr f R,4 and Perform feature optimization to enhance the authenticity-related illumination information and suppress the data domain-related illumination information. The heavy lighting model F RL Use the existing open source re-lighting model implementation, whose model weights are frozen and not involved in training, only used for generation The contrast loss L ctr The calculation formula is as follows: Among them, E(·) is used to measure the label consistency of the feature, label(·) represents the true and false labels corresponding to the sample features, Sim(·) represents the negative cosine similarity, and ‖·‖2 represents the L2 norm.

6. The face-swap video detection method based on illumination feature decoupling according to claim 1 is characterized in that: The illumination invariant feature f output by the last layer of the dual-branch high semantic feature extraction module is I,4 Send it to the Adversarial Learning Module (ALM) and use the adversarial loss L adv Enhance the domain invariance of features. The specific steps include: The illumination invariant feature f output by the last layer of the dual-branch high semantic feature extraction module is I,4 Send it to ALM for feature optimization and use adversarial loss L adv Enhance the domain-invariant properties of features. The adversarial loss L adv The calculation formula is as follows: where X and Y D Represent the sample set and domain label set respectively, N D is the number of data domains. 1[i=y] means that when the domain number i is equal to the domain type prediction result y, the output value is 1, otherwise it is 0. G represents the generation of f I,4 The feature extraction network of , D represents the domain discriminator. In order to optimize the generator G and D simultaneously, this method uses the Gradient Reversal Layer (GRL) to multiply the gradient by a negative scalar during back propagation to achieve gradient reversal.

7. The face-swap video detection method based on illumination feature decoupling according to claim 1 is characterized in that: The hierarchical feature fusion module (HFFM) is used to fuse the features of different levels in the dual-branch feature extraction network to obtain the high-semantic classification feature f A , the specific steps include: HFFM is used to fuse the features of different levels in the dual-branch feature extraction network to obtain the highly semantic classification feature f A . The f A The calculation formula is as follows: Where k represents the network level of the dual-stream high semantic feature extraction module, k = [1, 4], f A,k represents the features after the k-th layer network fusion, f A,0 =0. represents the HFFM operation of the kth layer, Represents a two-dimensional convolution operation, which is used to align the channel dimensions of features at different levels. Represents an upsampling operation, which is used to align the spatial dimensions of features at different levels for element-wise addition.

8. The face-swap video detection method based on illumination feature decoupling according to claim 1 is characterized in that: The classification feature f A Enter the classifier to get the predicted probability distribution, and use the cross entropy loss L cls Perform binary classification supervision. The specific steps include: The classification feature f A Enter the classifier to get the predicted probability distribution p, and use the cross entropy loss function L cls To conduct supervision; The cross entropy loss L cls The calculation formula is as follows: where N C Indicates the number of categories, that is, N C =2, p and y represent the predicted probability and classification label respectively.

9. The face-swap video detection method based on illumination feature decoupling according to claim 1, characterized in that: The model training updates the network parameters with the goal of minimizing the total loss function. After the training is completed, the network model and parameters are saved. The specific steps are as follows: Model training updates the network parameters with the goal of minimizing the total loss function, and saves the network model and parameters after training is completed. The total loss function L is as follows: L=L cls +α·L ctr +β·L adv Among them, α and β are hyperparameters that balance the two loss functions.

10. The face-swap video detection method based on illumination feature decoupling according to claim 1, characterized in that: The model test loads the trained detection model and outputs the detection results of the video to be tested.

11. A face-swap video detection method based on illumination feature decoupling based on the detection method according to any one of claims 1 to 10, characterized in that: include: Data preprocessing module, learnable local gravity pattern feature extraction module, dual-branch high semantic feature extraction module, contrastive learning module based on re-illumination, adversarial learning module, hierarchical feature fusion module, binary classification supervision module, training module and testing module; The data preprocessing module is used to decode the video into a frame sequence and extract the RGB image of each frame detection area frame by frame. RGB ; The learnable local gravity pattern feature extraction module is used to extract the I RGB The LPLGF feature extraction module is used to extract the local gravity pattern image I. LPLGF ; The dual-branch high semantic feature extraction module is used to RGB and I LPLGF Send it to the dual-stream high semantic feature extraction module to extract the illumination-related features f R,k and illumination-invariant features f I,k , k represents the level of the feature extraction network used, k = [1,4]; The contrastive learning module based on re-illumination is used to transform the illumination-related features f output by the last layer of the dual-branch high semantic feature extraction module into R,4 Send it to the Relighting-based Contrastive Learning Module (RCLM) and use the contrast loss L ctr Enhance the illumination information related to authenticity and suppress the illumination information related to the data domain; The adversarial learning module is used to transform the illumination invariant feature f output by the last layer of the dual-branch high semantic feature extraction module I,4 Send it to the Adversarial Learning Module (ALM) and use the adversarial loss L adv Enhance the domain-invariant properties of features; The hierarchical feature fusion module is used to fuse the features of different levels in the dual-branch feature extraction network using the hierarchical feature fusion module (HFFM) to obtain a high-semantic classification feature f A ; The two-class supervision module is used to classify the classification feature f A Enter the classifier to get the predicted probability distribution, and use the cross entropy loss L cls Conduct binary supervision; The training module is used for model training, updates network parameters with the goal of minimizing the total loss function, and saves the network model and parameters after training is completed; The test module is used for model testing, loading the trained detection model, and outputting the detection results of the video to be tested.

Images