Face forgery detection method and system based on reconstruction learning and hybrid expert mode

Abstract

The invention provides a face forgery detection method and system based on reconstruction learning and a hybrid expert mode. The method comprises the following steps: extracting a face frame image from a face video; performing data processing on the face frame image to obtain a two-dimensional face image; swin Transform is used as an encoder, a hybrid expert model is introduced at the same time, and spatial feature and long-distance dependence feature extraction is carried out on the two-dimensional face image; enhancing different types of features in the two-dimensional face image by using a hybrid expert model to obtain a high-level semantic feature map; generating a face reconstruction image by using an attention-guided decoding module; and aggregating the face reconstruction image and the corresponding two-dimensional face image by using a feature aggregation module based on reconstruction guidance, and exploring a possible counterfeit region to obtain a final feature map for face counterfeit detection. According to the method, the detection capability of unknown counterfeit types is better, and the accuracy, robustness and generalization of face counterfeit detection can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of face forgery detection, and in particular to a face forgery detection method and system based on reconstruction learning and hybrid expert mode. Background Art

[0002] Face forgery detection plays a vital role in maintaining social stability and national security. Initially, face forgery detection was mainly achieved by manually designing complex manual features. However, this method is inefficient and the results are unsatisfactory when dealing with more realistic face forgery images. Therefore, using deep learning technology to replace manual design has gradually become a mainstream trend. This model has the characteristics of high efficiency and low cost, and has been widely used in the field of face forgery detection. At present, although there are some methods that can achieve efficient face forgery detection, these methods are not perfect. Therefore, how to further improve the accuracy of face forgery detection has become an urgent problem to be solved.

[0003] Currently, deep learning-based face forgery detection faces the following three challenges: (1) Face deep forgery technology is constantly evolving, resulting in a gap between current detection technology and frequently iterated forged image or video generation technology. Although the detection accuracy for known forgery types is high, the performance for unknown new forgery types is significantly degraded. (2) Even if a model performs well in a single scenario, its performance will drop sharply when faced with complex situations such as compression and noise that are common in real scenarios. (3) Existing face forgery detection methods usually rely on expanding the model architecture to improve detection accuracy. However, the large number of parameters and high computational complexity make the model extremely difficult to deploy in practice.

[0004] With the development of deep learning, reconstruction learning has been widely used for representation learning in unsupervised environments. It encourages the model to encode more information about the input, and it can effectively restore the input information by modeling the distribution of input data in the embedding space. At the same time, the hybrid expert model has been widely used in computer vision and natural language processing. It uses a gating mechanism to dynamically select the most relevant Top-k experts, increase the capacity of the model without increasing computational overhead, and thus improve the accuracy and generalization of the model.

[0005] Currently, there is a lack of research on face forgery detection using reconstruction learning, hybrid expert models, attention graph calculation and feature aggregation methods. Summary of the invention

[0006] The purpose of the present invention is to provide a method and system for face forgery detection based on reconstruction learning and hybrid expert mode, so as to improve the accuracy, robustness and generalization of face forgery detection.

[0007] To achieve the above object, the present invention provides a method for detecting face forgery based on reconstruction learning and hybrid expert mode in a first aspect, the method comprising the following steps: Extract face frame images from face videos; Performing data processing on the face frame image to obtain a two-dimensional face image; Using Swin Transformer as an encoder, a hybrid expert model is introduced to extract spatial features and long-distance dependency features from two-dimensional face images; The hybrid expert model is used to enhance different types of features in two-dimensional face images to obtain high-level semantic feature maps; Using the attention-guided decoding module, the high-level semantic feature maps of different face images are calculated to suppress the decoding module from focusing on useless feature areas and generate face reconstruction images; Using the reconstruction-guided feature aggregation module, the reconstructed face image is aggregated with the corresponding two-dimensional face image to explore possible forgery areas and obtain the final feature map for face forgery detection.

[0008] Furthermore, the data processing of the face frame image includes image denoising, contrast enhancement, image scaling, image rotation, and image horizontal and vertical mirror transformations, and the image denoising includes dilation, erosion, and median filtering operations.

[0009] Furthermore, the encoder is composed of a plurality of encoder blocks, and each encoder block at a different level includes an LN module, a hybrid expert model, a W-MSA module and a SW-MSA module.

[0010] Furthermore, the input of the hybrid expert model is a face self-attention feature map, which is obtained by extracting spatial features and long-distance dependency features of the two-dimensional face image by the LN module, W-MSA module and SW-MSA module in the encoder block. Each expert sub-network in the hybrid expert model learns the feature information corresponding to the two-dimensional face image, and automatically selects different LoRA modules and adapter modules based on the gating mechanism to enhance different types of features in the two-dimensional face image to obtain a high-level semantic feature map.

[0011] Furthermore, the calculation method of the attention-guided decoding module is as follows: ,

[0012] In the above formula, and They represent depth-wise separable convolution and ordinary convolution layers with a convolution kernel size of 3×3, respectively. represents the sigmoid activation function, represents element-wise multiplication, Represents the feature map after the series operation in the channel dimension, and Respectively represent the attention feature map and the face reconstructed image output by the attention-guided decoding module.

[0013] Furthermore, the input of the reconstruction-guided feature aggregation module is: pixel-level difference encoding between the two-dimensional face image and the face reconstructed image and a high-level semantic feature map, and the feature aggregation module includes channel-aware global average pooling, downsampling, and convolution operations of two convolution kernel sizes.

[0014] A second aspect of the present invention provides a face forgery detection system based on reconstruction learning and hybrid expert mode, the system comprising: A face image extraction unit, used for extracting a face frame image from a face video; A face image preprocessing unit, used for performing data processing on the face frame image to obtain a two-dimensional face image; The feature encoding unit is used to use Swin Transformer as an encoder and introduce a hybrid expert model to extract spatial features and long-distance dependent features from two-dimensional face images; A hybrid expert feature enhancement unit is used to enhance different types of features in a two-dimensional face image to obtain a high-level semantic feature map; A reconstructed image decoding generation unit is used to use the attention-guided decoding module to perform attention calculation on the high-level semantic feature maps of different face images, suppress the decoding module from focusing on useless feature areas, and generate a face reconstructed image; The reconstruction-guided feature aggregation unit is used to aggregate the face reconstruction image with the corresponding two-dimensional face image using the feature aggregation module based on reconstruction guidance, explore possible forgery areas, and finally obtain a feature map for face forgery detection.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention introduces a hybrid expert model in the encoder, dynamically selects the LoRA module and the adapter module through a gating mechanism, increases the model capacity without increasing the computational overhead, and thus improves the accuracy and generalization of the model; 2. The present invention uses an attention-guided decoding module to perform attention calculation on high-level semantic feature maps of different face images, which can enhance the model's ability to perceive subtle features, inhibit the model from focusing on useless feature areas, and generate realistic and more compact face reconstruction images; 3. The present invention aggregates the face reconstructed image with the corresponding two-dimensional face image, so that the model can further explore the possible forged areas in the face reconstructed image, thereby improving the model's ability to detect unknown forgery types. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only preferred embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 It is a flowchart of a face forgery detection method based on reconstruction learning and hybrid expert mode provided by an embodiment of the present invention.

[0018] Figure 2 It is a schematic diagram of a flow chart of data processing of a face frame image provided by an embodiment of the present invention.

[0019] Figure 3 It is a schematic diagram of the operating principle of an encoder based on Swin Transformer provided in an embodiment of the present invention.

[0020] Figure 4 It is a schematic diagram of the operating principle of the feature aggregation module based on reconstruction guidance provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Reference Figure 1 This embodiment provides a face forgery detection method based on reconstruction learning and hybrid expert mode to improve the accuracy, robustness and generalization of face forgery detection. The method includes the following steps: S100: extracting face frame images from face videos.

[0023] S200: performing data processing on the face frame image to obtain a two-dimensional face image. Figure 2 A process of performing data processing on a face frame image provided by an embodiment of the present invention is shown.

[0024] Exemplarily, the data processing includes but is not limited to image denoising, contrast enhancement, image scaling, image rotation, and image horizontal and vertical mirror transformation. The image denoising includes at least three operations: dilation, erosion, and median filtering. This embodiment processes the face frame image data to meet the training requirements of deep learning in terms of data quantity and data diversity.

[0025] S300 uses Swin Transformer as an encoder and introduces a hybrid expert model to extract spatial features and long-distance dependent features from two-dimensional face images. Figure 3 The operating principle of an encoder based on SwinTransformer provided by an embodiment of the present invention is shown.

[0026] Affected by factors such as shooting equipment and environment, when obtaining face frame images from face videos, they are often affected by factors such as video or image resolution or compression quality. Therefore, extracting key features and forged area information from the acquired two-dimensional face images can enhance the model's ability to resist adverse factors and make the encoder suitable for processing face image data with complex spatial structures and long-term dependencies.

[0027] As an implementation of step S300, an encoder based on Swin Transformer is composed of multiple encoder blocks, and each encoder block at different levels includes an LN module, a hybrid expert model, a W-MSA (Window-based Multi-Head Self-Attention) module and a SW-MSA (Shifted Window-based Multi-Head Self-Attention) module. The encoder allows the model to focus on different parts of the face image through multiple layers of self-attention mechanisms and feedforward neural networks, thereby effectively aggregating information across image blocks, and processing image block embedding in a hierarchical manner to capture local and global dependencies in the image. In addition, by introducing hybrid expert models in encoder blocks at different levels, it is possible to effectively extract spatial features and long-distance dependency features from two-dimensional face images, and process image block embedding in a hierarchical manner to capture local and global dependencies in the image.

[0028] Assume l The output of W-MSA and SW-MSA of blocks is expressed as and , then the calculation process of its encoder is defined as follows:

[0029]

[0030]

[0031]

[0032] In the above formula, W-MSA, LN, MoE and SW-MSA represent the window-based multi-head self-attention layer, normalization layer, mixture of expert model layer and shifted window multi-head self-attention layer, respectively.

[0033] S400, using a hybrid expert model to enhance different types of features in a two-dimensional face image to obtain a high-level semantic feature map.

[0034] Although the encoder can effectively extract shallow and deep semantic information of face images, the forged regions in face images have irregular shapes, diverse scales, and blurred boundaries, which poses a great challenge to forged face image detection. In order to better cope with these challenges, the method provided in this embodiment specifically introduces a hybrid expert model in the encoder block, aiming to enhance different types of features in the image.

[0035] Mixture of Experts (MoE) is a model architecture widely used in the fields of machine learning and deep learning. It improves the efficiency and performance of the model by combining multiple expert sub-networks, each of which focuses on processing a specific subset of input data. The core idea of ​​the MoE model is to use a "gating network" to determine which expert models should be activated to process specific input data. This method can not only improve the expressiveness of the model, but also reduce the consumption of computing resources to a certain extent. However, how to assign weights to the output of each expert sub-network is the main challenge in the design of this model. Unlike the traditional mixed expert model, this embodiment uses a gating mechanism to select different LoRA modules and adapter modules to improve the sensitivity and accuracy of features of different scales.

[0036] As an implementation method of step S400, the input of the hybrid expert model is a face self-attention feature map, which is obtained by extracting spatial features and long-distance dependency features of a two-dimensional face image by the LN module, the W-MSA module, and the SW-MSA module in the encoder block. Each expert subnetwork in the hybrid expert model learns feature information corresponding to the two-dimensional face image, and automatically selects different LoRA (Low-Rank Adaptation) modules and adapter modules based on a gating mechanism to enhance different types of features in the two-dimensional face image to obtain a high-level semantic feature map. The gating mechanism enables the model to dynamically allocate computing resources according to the complexity of the image and enhance the representation ability of the model.

[0037] The LoRA module is implemented by freezing the pre-trained weight matrix during training and adding the product of two trainable low-rank matrices. The adapter module is composed of several trainable convolutional layers. The forward propagation process of the LoRA module is defined as follows:

[0038] in, W represents the weight matrix of the pre-trained ImageNet, represents the product of two trainable low-rank matrices added by the LoRA module ( ), x Represents the input data of the model.

[0039] The calculation method of the adapter module is defined as follows:

[0040] In the above formula, and They represent the multi-layer perceptron layer and bypass convolutional layer respectively. and Represent the output characteristics and input characteristics of the adapter module respectively.

[0041] Exemplarily, the weight calculation method of each expert sub-network is defined as follows:

[0042] in, Represents the gating weight of the input data x after normalization by the SoftMax function, and Represent the unnormalized input data x The gating weight and j The gating weight of each expert is represents the sum of the gating weights of all experts in the current expert network, and Represent the SoftMax function and the natural exponential function respectively, and denote the learnable weights and biases of the gating network, respectively.

[0043] S500, using the attention-guided decoding module to perform attention calculation on high-level semantic feature maps of different face images, suppressing the decoding module from focusing on useless feature areas, and generating a face reconstructed image.

[0044] As an implementation method of step S500, the attention-guided decoding module includes operations such as depthwise separable convolution (DW) and skip connection (SC). The attention-guided decoding module is used in the Swin Transformer block and a high-level semantic feature map is extracted to obtain an attention map with the same shape as the input feature, and the decoding module is suppressed from focusing on useless feature areas to generate a realistic and more compact face reconstruction image.

[0045] In this optional implementation, the attention-guided decoding module first takes the output of the previous attention-guided decoding module and the high-level semantic feature map of the corresponding level in the backbone network as input, performs a concatenation operation on the channel dimension, and then performs a depth-separable convolution operation , and obtain an attention feature map with the same shape as the input feature , the corresponding calculation formula is:

[0046]

[0047] In the above formula, 、 and They represent the feature map after the concatenation operation, the output features of the previous attention-guided decoding module, and the high-level semantic feature map of the corresponding level in the backbone network, respectively. and They represent the channel dimension feature concatenation function and the sigmoid activation function respectively.

[0048] Exemplarily, the computational method of the attention-guided decoding module is defined as follows: ,

[0049] In the above formula, and They represent depth-wise separable convolution and ordinary convolution layers with a convolution kernel size of 3×3, respectively. represents the sigmoid activation function, represents element-wise multiplication, Represents the feature map after the series operation in the channel dimension, and Respectively represent the attention feature map and the face reconstructed image output by the attention-guided decoding module.

[0050] S600: Utilizing a reconstruction-guided feature aggregation module, the face reconstruction image is aggregated with the corresponding two-dimensional face image to explore possible forgery areas, and finally obtain a feature map for face forgery detection. Figure 4 The operating principle of a feature aggregation module based on reconstruction guidance provided by an embodiment of the present invention is shown.

[0051] As is known to all, different face images usually contain different image information. Therefore, in order to achieve good integration and obtain a realistic and compact visual representation, this embodiment uses pixel-level difference masks and encoder features for fusion interaction to mine the difference clues between the reconstructed visual representation and the real visual representation.

[0052] As an implementation of step S600, the input of the reconstruction-guided feature aggregation module is: pixel-level difference encoding between the two-dimensional face image and the face reconstruction image and a high-level semantic feature map. and 2D face images , first calculate the pixel-level difference between the two , expressed as:

[0053] The feature aggregation module includes channel-aware global average pooling (GAP), downsampling, and convolution operations with two convolution kernel sizes (1×1 and 3×3). By aggregating the reconstructed image and the face image, it is possible to further explore the possible forged areas in the face reconstructed image, and improve the model's ability to detect unknown forgery types. In order to fuse pixel-level difference encoding with high-level semantic features and enhance feature representation, this optional implementation designs a channel-aware global average pooling, and then obtains the corresponding channel weights through one-dimensional convolution and sigmoid activation function.

[0054] The calculation methods of fusion features and output aggregation features are respectively expressed as:

[0055]

[0056] In the above formula, represents the downsampling operation, represents the global average pooling operation at the channel level, represents the fused features, represents the aggregated features of the output, and They represent convolution with a kernel size of 1×1 and one-dimensional convolution operations respectively. and Represent the sigmoid activation function and channel-aware global average pooling operation, is an element-by-element multiplication operation, represents element-wise addition, represents the high-level semantic features of the encoder, Represents a convolution operation with a convolution kernel size of 3×3.

[0057] Based on the above operations, by utilizing the reconstruction-guided feature aggregation module, the real and more compact face reconstruction image is aggregated with the two-dimensional face image to explore possible forgery areas, and finally obtain the feature map used for face forgery detection.

[0058] Another embodiment of the present invention further provides a face forgery detection system based on reconstruction learning and hybrid expert mode, the system is used to perform the method steps described in the above method embodiment. The system includes: A face image extraction unit, used for extracting a face frame image from a face video; A face image preprocessing unit, used for performing data processing on the face frame image to obtain a two-dimensional face image; The feature encoding unit is used to use Swin Transformer as an encoder and introduce a hybrid expert model to extract spatial features and long-distance dependent features from two-dimensional face images; A hybrid expert feature enhancement unit is used to enhance different types of features in a two-dimensional face image to obtain a high-level semantic feature map; A reconstructed image decoding generation unit is used to use the attention-guided decoding module to perform attention calculation on the high-level semantic feature maps of different face images, suppress the decoding module from focusing on useless feature areas, and generate a face reconstructed image; The reconstruction-guided feature aggregation unit is used to aggregate the face reconstruction image with the corresponding two-dimensional face image using the feature aggregation module based on reconstruction guidance, explore possible forgery areas, and finally obtain a feature map for face forgery detection.

[0059] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0060] This article uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only used to help understand the core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A face forgery detection method based on reconstruction learning and hybrid expert mode, characterized in that: The method comprises the steps of: Extract face frame images from face videos; Performing data processing on the face frame image to obtain a two-dimensional face image; Using Swin Transformer as an encoder, a hybrid expert model is introduced to extract spatial features and long-distance dependency features from two-dimensional face images; The hybrid expert model is used to enhance different types of features in two-dimensional face images to obtain high-level semantic feature maps; Using the attention-guided decoding module, the high-level semantic feature maps of different face images are calculated to suppress the decoding module from focusing on useless feature areas and generate face reconstruction images; Using the reconstruction-guided feature aggregation module, the reconstructed face image is aggregated with the corresponding two-dimensional face image to explore possible forgery areas and obtain the final feature map for face forgery detection.

2. The method according to claim 1, characterized in that: The data processing of the face frame image includes image denoising, contrast enhancement, image scaling, image rotation, and image horizontal and vertical mirror transformations. The image denoising includes dilation, erosion, and median filtering operations.

3. The method according to claim 1, characterized in that The encoder consists of multiple encoder blocks, and each encoder block at different levels includes an LN module, a mixed expert model, a W-MSA module and a SW-MSA module.

4. The method according to claim 3, characterized in that The input of the hybrid expert model is a face self-attention feature map, which is obtained by extracting spatial features and long-distance dependency features of a two-dimensional face image by the LN module, W-MSA module and SW-MSA module in the encoder block. Each expert sub-network in the hybrid expert model learns the feature information corresponding to the two-dimensional face image, and automatically selects different LoRA modules and adapter modules based on the gating mechanism to enhance different types of features in the two-dimensional face image to obtain a high-level semantic feature map.

5. The method according to claim 1, characterized in that The attention-guided decoding module is calculated as follows: , In the above formula, and They represent depth-wise separable convolution and ordinary convolution layers with a convolution kernel size of 3×3, respectively. represents the sigmoid activation function, represents element-wise multiplication, Represents the feature map after the series operation in the channel dimension, and Respectively represent the attention feature map and the face reconstructed image output by the attention-guided decoding module.

6. The method according to claim 1, characterized in that The input of the reconstruction-guided feature aggregation module is: pixel-level difference encoding between a two-dimensional face image and a face reconstruction image and a high-level semantic feature map. The feature aggregation module includes channel-aware global average pooling, downsampling, and convolution operations of two convolution kernel sizes.

7. A face forgery detection system based on reconstruction learning and hybrid expert mode, characterized in that: The system comprises: A face image extraction unit, used for extracting a face frame image from a face video; A face image preprocessing unit, used for performing data processing on the face frame image to obtain a two-dimensional face image; The feature encoding unit is used to use Swin Transformer as an encoder and introduce a hybrid expert model to extract spatial features and long-distance dependent features from two-dimensional face images; A hybrid expert feature enhancement unit is used to enhance different types of features in a two-dimensional face image to obtain a high-level semantic feature map; A reconstructed image decoding generation unit is used to use the attention-guided decoding module to perform attention calculation on the high-level semantic feature maps of different face images, suppress the decoding module from focusing on useless feature areas, and generate a face reconstructed image; The reconstruction-guided feature aggregation unit is used to aggregate the face reconstruction image with the corresponding two-dimensional face image using the feature aggregation module based on reconstruction guidance, explore possible forgery areas, and finally obtain a feature map for face forgery detection.

Images