A dual-branch face spoofing detection method based on dual domains
By constructing a dual-domain, dual-branch network model, combining spatial and frequency domain features, and capturing boundary forgery clues and interactive features, the problems of insufficient detection and poor generalization ability in low-quality video environments are solved, achieving more efficient face forgery detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI UNIV
- Filing Date
- 2023-05-19
- Publication Date
- 2026-07-03
AI Technical Summary
Existing face forgery detection methods have insufficient detection performance and poor generalization ability in low-quality video environments, and cannot effectively identify compressed forged videos.
A dual-branch network model based on dual domains is adopted, which combines spatial and frequency domain features. The boundary supervision module captures boundary forgery clues, and the difference feature attention module interacts with frequency and spatial domain features to improve detection performance.
Improve detection performance and enhance the model's generalization ability in environments where video quality is compressed, effectively identifying low-quality fake videos.
Smart Images
Figure CN116597491B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of face detection technology, and in particular to a dual-domain, dual-branch face forgery detection method. Background Technology
[0002] In recent years, the rapid development of artificial intelligence technology has driven the continuous advancement of deepfake technology, enabling increasingly realistic fake face video images. Currently, the main popular face spoofing techniques are face replacement and face reconstruction. DeepFakes is the most representative face replacement technique, primarily based on an autoencoder. This autoencoder structure makes face swapping more convenient and efficient, significantly reducing technical requirements, and the relevant algorithms are already open-source on the internet. Subsequently, the emergence of technologies such as FSGAN and FaceShifter has greatly improved the quality of face replacement. Face reconstruction technology involves forging facial attributes, such as facial expressions. Face2Face and NeuralTextures are representative algorithms in face reconstruction, achieving the transfer of expressions from the original person to the target person. Deepfakes are low-cost and easy to operate, eliminating the need for specific hardware or equipment, making large-scale participation by internet users possible. However, malicious individuals can also use this technology to spread fake videos for illegal activities such as commercial defamation and extortion, and even exacerbate social conflicts, posing a significant threat to personal and public safety.
[0003] Therefore, effective deepfake detection methods are a crucial tool for maintaining the network environment and an important measure to enhance the credibility of cyberspace, playing a vital role in protecting individual rights and national security.
[0004] Because fake video images and real video images have inconsistencies in features from multiple angles, current face forgery detection methods distinguish the authenticity of faces from different perspectives.
[0005] Because the human face is composed of five facial features whose shape and positional relationships are relatively unchanging, and because a person exhibits a series of physiological characteristics during a speech due to physical condition, environment, and other factors, these biological characteristics are highly likely to be overlooked during face forgery, leading to inconsistencies in the forged face. This characteristic has been widely used in face forgery detection, utilizing the biological characteristics of the face for forgery detection. However, with the continuous improvement of face forgery technology, this type of detection method has gradually become ineffective.
[0006] Another approach is to use convolutional neural networks for face forgery detection, continuously improving the network's detection performance by training it with a large number of real and fake faces. However, this method has poor generalization performance, and its detection performance drops significantly when faced with unknown forgery techniques. Furthermore, its detection performance is also noticeably poor when dealing with compressed, low-quality videos. In summary, existing face detection methods have the following drawbacks: insufficient detection performance in environments with compressed video quality, and poor generalization ability. Summary of the Invention
[0007] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a dual-domain dual-branch face forgery detection method. This method is applied to the field of face forgery detection and overcomes some limitations of existing face forgery detection schemes, namely, the inability to effectively detect low-quality face forgery videos. In addition, this invention also proposes a dual-domain dual-branch framework to improve the general performance of the model.
[0008] The objective of this invention can be achieved through the following technical solutions:
[0009] A dual-domain, dual-branch face forgery detection method includes the following steps:
[0010] Construct a dataset of videos featuring fake faces;
[0011] The video data in the aforementioned fake face video dataset is preprocessed to obtain preprocessed face images;
[0012] Based on the preprocessed face images, the dataset is divided into training, validation, and test sets, respectively.
[0013] Construct a dual-branch network model, train the dual-branch network model using the training set, and obtain the optimal model;
[0014] Using the optimal model, frequency domain features and spatial domain features of the preprocessed face image are extracted, and the frequency domain features and spatial domain features are interacted to obtain the face forgery detection result.
[0015] Furthermore, preprocessing the video data in the dataset includes the following steps:
[0016] The video data in the aforementioned fake face video dataset is divided into a set of continuous, non-overlapping video frames;
[0017] Face detection is performed on the video frames sequentially to extract face regions;
[0018] Remove background information outside the face area in the video frame.
[0019] Furthermore, the extraction of the spatial features includes the following steps:
[0020] Input the raw RGB image;
[0021] The texture feature information of the original RGB image is captured, and traces of forgery are found through contrast and color differences;
[0022] Boundary supervision is performed on the original RGB image, and forgery clues are mined through shallow boundary information to obtain spatial features.
[0023] Furthermore, the extraction of the frequency domain features includes the following steps:
[0024] Input the raw RGB image;
[0025] Obtain the frequency components of the original RGB image;
[0026] The frequency components are decomposed using an adaptive frequency filter to obtain low-frequency components, mid-frequency components, high-frequency components, and full-frequency components.
[0027] The divided frequency components are transformed into spatial components in the spatial domain.
[0028] The spatial components are arranged, and frequency domain features are extracted based on the arranged spatial components.
[0029] Furthermore, the dual-branch network model includes an information extraction module, a boundary supervision module, and a differential feature attention module;
[0030] The information extraction module is used to extract the frequency domain features and spatial domain features of the preprocessed face image;
[0031] The boundary supervision module is used to learn the boundary features of the image's spatial features and capture forgery clues at the boundary.
[0032] The differential feature attention module is used to interact with frequency domain features and spatial domain features to obtain face forgery detection results.
[0033] Furthermore, the boundary supervision module includes a Sobel layer and an attention module.
[0034] Furthermore, the difference feature attention module models the correlation between spatial and frequency domain features by calculating attention. Based on the learned feature information, it focuses on the forgery traces hidden in the preprocessed face image and uses the learned differences to determine the authenticity of the video image.
[0035] Furthermore, the face spoofing video dataset includes real videos and synthetic videos with different compression parameters.
[0036] Furthermore, this method also includes the following step: based on the validation set, selecting the best-performing dual-branch network model as the optimal model. Using a validation set helps to better select the best training model and prevents overfitting during training.
[0037] Furthermore, this method also includes the following steps: based on the test set, using the prediction results of the test set as the basis for evaluating the performance of the dual-branch network model.
[0038] Compared with the prior art, the present invention has the following beneficial effects:
[0039] (1) This invention uses a boundary supervision module to supervise the boundary of the image spatial domain, captures forgery clues at the boundary to detect forged face images, and improves the performance of the face forgery detection network.
[0040] (2) The dual-branch network model proposed in this invention captures forgery traces in the frequency domain features while focusing on the spatial features of the image, which can enhance the generalization performance to a certain extent and improve the detection performance in the environment where the video quality is compressed.
[0041] (3) By proposing a differential feature attention module, this invention is no longer limited to feature information in a single domain. It allows for interaction between spatial and frequency domain information, promoting feature learning between different domains and improving the detection performance of the network. Attached Figure Description
[0042] Figure 1 The flowchart shows a dual-domain, dual-branch face forgery detection method.
[0043] Figure 2 This is a diagram of the overall network structure used in the method of this invention.
[0044] Figure 3 This is a structural diagram of the boundary supervision module used in the method of this invention.
[0045] Figure 4 This is a structural diagram of the differential feature attention module used in the method of this invention embodiment. Detailed Implementation
[0046] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.
[0047] This invention is based on deep learning technology and constructs a dual-branch network model, such as... Figure 2As shown, the network model comprises three modules: an information extraction module, a boundary supervision module, and a differential feature attention module. Starting from the frequency and spatial domains of the image, it learns the image's color and texture information simultaneously with its frequency domain information. Specifically, for the spatial domain information, it focuses on learning boundary features to capture forgery clues at the boundaries. Combined with the attention module, it focuses on hidden forgery traces in face images based on the learned feature information, and uses the learned differential features to distinguish between real and fake video images.
[0048] Based on the above inventive concept, the present invention adopts the following technical solution:
[0049] like Figure 1 As shown, a dual-domain, dual-branch face forgery detection method includes the following steps:
[0050] (1) Constructing a Face Forgery Video Dataset. This embodiment selects the open-source face forgery video dataset FaceForensics++. The FaceForensics++ dataset is the first large-scale face forgery video dataset. It contains three subsets with different compression levels: no compression (C0), light compression (C23), and heavy compression (C40), which is beneficial for better simulating videos of different qualities circulating on the Internet. The dataset includes 1,000 real videos and 5,000 synthetic videos selected from YouTube. The synthetic videos are generated by four forgery techniques: DeepFakes, FaceSwap, Face2Face, and NeuralTextures. The first two forgery techniques belong to facial identity replacement, and the latter two belong to facial expression reconstruction.
[0051] (2) The video data in the face spoofing video dataset is preprocessed to obtain preprocessed face images. First, in this embodiment, the OpenCV tool library is used to divide all real or spoofed face videos into a set of continuous non-overlapping video frames, capturing 15 frames per second. Then, the Dlib toolkit is used for face detection, extracting the face region, aligning the face, and removing a large amount of irrelevant background information to prevent background information from affecting the accuracy of face spoofing detection, because face spoofing technology targets and tamperes with the face region and does not concern itself with the background region. Subsequently, the size of the aligned image is adjusted to 256×256.
[0052] (3) Based on the preprocessed face images, the dataset is divided into training, validation, and test sets. In this embodiment, the face images obtained in the previous step are divided in a ratio of 7:1.5:1.5 for training, validation, and testing tasks, i.e., the training set accounts for 70%, the validation set accounts for 15%, and the test set accounts for 15%. Using the validation set helps to better select the best training model and prevent overfitting during training. The prediction results of the test set are used as the basis for evaluating the performance of the network model.
[0053] (4) Construct a dual-branch network model and train the dual-branch network model using the training set to obtain the optimal model. In order to effectively detect face forgery, this invention proposes a dual-branch network based on two domains, including spatial domain flow and frequency domain flow.
[0054] like Figure 2 As shown, the dual-branch network model includes an information extraction module, a boundary supervision module, and a differential feature attention module;
[0055] The information extraction module is used to extract the frequency domain features and spatial domain features of the preprocessed face image; the information extraction module includes feature extraction of different branches, namely image spatial domain feature extraction and image frequency domain feature extraction.
[0056] like Figure 3 As shown, the boundary supervision module is used to learn the boundary features of the image's spatial domain features and capture forgery clues at the boundaries. The boundary supervision module consists of a Sobel layer and an attention module, which helps the network effectively supervise the boundaries in the original RGB image and deeply mine forgery clues through shallow boundary information.
[0057] like Figure 4 As shown, the differential feature attention module interacts with frequency domain features and spatial domain features to obtain face forgery detection results. The differential feature attention module models the correlation between spatial and frequency domain information by calculating attention, focuses on hidden forgery traces in face images based on the learned feature information, and uses the learned differences to distinguish between real and fake video images.
[0058] In the information extraction module, the input to the spatial domain stream is the original RGB image. The backbone network fully captures the image's texture features, identifying forgery traces through contrast and color differences. Simultaneously, the boundary supervision module helps the network effectively supervise the boundaries in the original RGB image, using shallow boundary information to deeply mine forgery clues. The frequency domain stream first processes the original RGB image. Specifically, it obtains the frequency components from the original RGB image and then uses a set of adaptive frequency filters to decompose the frequency components into low-frequency, mid-frequency, high-frequency, and full-frequency components. The full-frequency component refers to the entire frequency domain, and its purpose is to prevent the loss of subtle forgery clues due to frequency component segmentation. The segmented frequency components are transformed into spatial components, arranged along the channels, and then input into the backbone network to fully extract frequency domain information.
[0059] In this embodiment, video preprocessing mainly consists of two steps. The first step is to extract frames from the video. Using the OpenCV video processing library, the video data is cropped into frame images at a rate of 15 frames per second. This increases the amount of image data, making it easier for the network model to train. The second step is to locate and extract faces from the frame images. Since there is a lot of irrelevant background information around the face in the image, and face spoofing technology targets the face region and is unrelated to background information, it is necessary to obtain the smallest possible face bounding box using a face localization algorithm to avoid background information affecting the network detection performance. This face bounding box is then expanded as a new image region and saved as the final input image.
[0060] In summary, this invention proposes a dual-domain, dual-branch face forgery detection method, comprising an information extraction module, a boundary supervision module, and a differential feature attention module. The information extraction module searches for forgery traces in both the spatial and frequency domains of the image in parallel. The boundary supervision module encourages the network model to pay closer attention to subtle false traces at facial boundaries. The differential feature attention module, based on an attention mechanism, focuses on the differences in features, ignoring unimportant information and promoting the interaction and fusion of frequency and spatial domain information. Finally, the deep features are input into a classifier to distinguish between genuine and fake face images. This method addresses the shortcomings of existing technologies, such as insufficient detection performance and poor generalization ability in environments with compressed video quality.
[0061] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.
Claims
1. A dual-domain based dual-branch flow face forgery detection method, characterized in that, Includes the following steps: Construct a dataset of videos featuring fake faces; The video data in the aforementioned fake face video dataset is preprocessed to obtain preprocessed face images; Based on the preprocessed face images, the dataset is divided into training, validation, and test sets, respectively. Construct a dual-branch network model, train the dual-branch network model using the training set, and obtain the optimal model; Using the optimal model, frequency domain features and spatial domain features of the preprocessed face image are extracted, and the frequency domain features and spatial domain features are interacted to obtain the face forgery detection result; The dual-branch network model includes an information extraction module, a boundary supervision module, and a differential feature attention module. The information extraction module is used to extract the frequency domain features and spatial domain features of the preprocessed face image; The boundary supervision module is used to learn the boundary features of the image's spatial features and capture forgery clues at the boundary. The differential feature attention module is used to interact with frequency domain features and spatial domain features to obtain face forgery detection results; The differential feature attention module models the correlation between spatial and frequency domain features by calculating attention. Based on the learned feature information, it focuses on the forgery traces hidden in the preprocessed face image and uses the learned differences to determine the authenticity of the video image.
2. The dual-domain based dual-branch face forgery detection method according to claim 1, wherein, Preprocessing the video data in the dataset includes the following steps: The video data in the aforementioned fake face video dataset is divided into a set of continuous, non-overlapping video frames; Face detection is performed on the video frames sequentially to extract face regions; Remove background information outside the face area in the video frame.
3. The dual-domain based dual-branch face forgery detection method according to claim 1, wherein, The extraction of the spatial features includes the following steps: Input the raw RGB image; The texture feature information of the original RGB image is captured, and traces of forgery are found through contrast and color differences; Boundary supervision is performed on the original RGB image, and forgery clues are mined through shallow boundary information to obtain spatial features.
4. The dual-domain based dual-branch face forgery detection method according to claim 1, wherein, The extraction of the frequency domain features includes the following steps: Input the raw RGB image; Obtain the frequency components of the original RGB image; The frequency components are decomposed using an adaptive frequency filter to obtain low-frequency components, mid-frequency components, high-frequency components, and full-frequency components. The divided frequency components are transformed into spatial components in the spatial domain. The spatial components are arranged, and frequency domain features are extracted based on the arranged spatial components.
5. The dual-domain, dual-branch face forgery detection method according to claim 1, characterized in that, The boundary supervision module includes a Sobel layer and an attention module.
6. The dual-domain based dual-branch face forgery detection method according to claim 1, wherein, The face spoofing video dataset includes real videos and synthetic videos with different compression parameters.
7. The dual-domain based dual-branch face forgery detection method according to claim 1, wherein, It also includes the following steps: Based on the validation set, the best-trained dual-branch network model is selected as the optimal model.
8. The dual-domain based dual-branch face forgery detection method according to claim 1, characterized in that, It also includes the following steps: Based on the test set, the prediction results of the test set are used as the basis for evaluating the performance of the dual-branch network model.