Deformable face spoofing detection network and spatiotemporal consistent face spoofing model construction method

Abstract

This invention relates to the field of face authentication technology, and discloses a deformable face authentication network and a spatiotemporally consistent face authentication model construction method. The deformable face authentication network includes a backbone network and a deformable temporal self-attention network. The backbone network is used to extract the spatial features of the face in the input image, and the deformable temporal self-attention network is used to process the spatial features of the face output by the backbone network to extract the temporal features of the face. The spatiotemporally consistent face authentication model construction method, based on the deformable face authentication network and the temporal self-attention feature mechanism, can alleviate the interference of semantic feature offset between face video frames on the extraction of temporal forgery features, thereby effectively improving the generalization ability of the authentication model. By constructing a spatiotemporally consistent face authentication model, accurate separation and representation of the spatiotemporal features of real faces and forged faces are achieved, thereby effectively improving the robustness of the authentication model against unknown forged faces.

Description

Technical Field

[0001] This invention relates to the field of face authentication technology, specifically to a deformable face authentication network and a spatiotemporally consistent face authentication model construction method. Background Technology

[0002] In recent years, image / video generation and synthesis algorithms have emerged, especially in the areas of full-image face generation, face replacement, facial attribute editing, and expression / pose replay, where the forgery effects are highly realistic. Deepfake face images / videos are difficult to effectively detect using the human eye or traditional techniques. Benefiting from the construction of forged face datasets such as FaceForensics++, Celeb-DF, DFDC, and FFIW, various deep learning-based face detection methods (called Deepfake Detection or FaceForgery Detection) have shown significant performance within individual datasets. However, due to the differences in forgery traces between different forged datasets, the generalization performance of detection models is poor. Recent research mainly focuses on how to improve the generalization ability of forgery detection algorithms. One approach is to design corresponding deep learning algorithms by analyzing the forgery clues hidden in synthesized images from the perspectives of color distortion, artifacts, GAN fingerprints, and high-frequency information. Another approach is to directly utilize deep learning strategies such as attention mechanisms, contrastive learning, and self-supervised learning to extract discriminative features in the spatial dimension of forged face image datasets.

[0003] Realistic forged face images exhibit subtle forgery signals within their spatial dimensions, and the significant differences in forgery patterns between algorithms severely limit the generalization ability of forgery detection models. In reality, most forgery algorithms (Deepfakes, FaceSwap, StyleGAN, etc.) only support frame-by-frame face editing or generation, making it difficult to avoid temporal differences in texture, lighting, and other appearance characteristics. Especially since the imaging mechanisms of real and forged face videos differ, generating temporally consistent forged face videos is extremely difficult. Furthermore, humans typically identify tampered facial features from both spatial and temporal dimensions. Therefore, mining the temporal feature differences between frames of forged face videos is key to improving the generalization ability of forgery detection models. Currently, a few methods are designed and constructed from the temporal perspective of face videos, the main methods being:

[0004] 1) Deepfake Video Detection with Spatiotemporal Dropout Transformer (ACM MM, 2022). Main technique: Randomly sample image patches from all frames of a face video, using these as input to a multi-layer self-attention transform (Transformer), and then use the feature vector from the category token as the discrimination result for the input video. Problems and drawbacks: The random sampling strategy at the input end of this method cannot fully utilize the complete information of the video; unused image patches may be key features, thus limiting the model's generalization ability.

[0005] 2) Exploring Temporal Coherence for More General Video Face Forgery Detection (ICCV, 2021). Main technical approach: Based on ResNet-3D, the spatial convolution kernel size of the 3D convolutional neural network is set to 1, while the temporal convolution kernel size remains unchanged, to constrain the network to focus on temporal feature differences in face videos. Problems and drawbacks: This method actually only extracts temporal features at the same spatial location between frames, without considering the semantic feature shift between frames caused by face movement. The temporal features extracted by this method are actually mixed with semantic shift features within the same location and temporal forgery features, and face shift features are often more obvious than temporal forgery features, thus seriously interfering with the extraction and discrimination of temporal forgery features, resulting in limited model performance.

[0006] Spatiotemporal Inconsistency Learning for DeepFake Video Detection (ACMM MM, 2021). Main techniques: Utilizing two attention mechanisms—spatial inconsistency and temporal inconsistency—to extract spatial and temporal forgery features from fake face videos. Problems and drawbacks: This method does not consider the impact of temporal feature shifts on forgery feature extraction, resulting in limited model detection accuracy. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the present invention aims to provide a deformable face authentication network and a spatiotemporally consistent face authentication model construction method.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A deformable face authentication network includes a backbone network and a deformable temporal self-attention network. The backbone network is used to extract face spatial features from the input image, and the deformable temporal self-attention network is used to process the face spatial features output by the backbone network to extract temporal features of the face.

[0010] In this invention, preferably, both the backbone network and the deformable temporal self-attention network are based on the Transformer model;

[0011] In this invention, preferably, the backbone network mainly performs the following operations:

[0012] S01, Extract blocks from the input image and encode the blocks to obtain a block sequence;

[0013] S02, Block sequence features for processing block sequences.

[0014] In this invention, preferably, the deformable temporal self-attention network mainly performs the following operations:

[0015] S11, Input block sequence features, calculate the similarity between all blocks by dot product, and obtain the set of blocks with the highest similarity for each block in other positions in the current frame and in other frames;

[0016] S12, calculate the cross-attention between blocks and block sets to obtain the updated block spatiotemporal feature encoding, i.e. the temporal features of the face.

[0017] In this invention, preferably, the deformable temporal self-attention network calculates the key point offset between each frame using the known facial key point coordinates, and uses this as a reference value to constrain the accuracy of the prediction of high similarity blocks of key points between frames. At the same time, the positional offset between the block where the key point is located and the most similar block in other frames can be calculated using other positions in the current frame and the block number with the highest similarity to the block in other frames.

[0018] A spatiotemporally consistent face authentication model construction method, preferably comprising the following steps:

[0019] S1, input the masked face video T-frame image, and obtain the reconstructed face video through the first model;

[0020] S2, the first model is improved to obtain the second model, which uses spatiotemporal feature extraction and enhancement of the input face video T-frame images;

[0021] S3, based on the second model, incorporates a multi-target supervision component to form a face authentication model, which is used to identify whether the input face video is real or fake.

[0022] In this invention, preferably, the first model includes a feature encoding module E, a spatiotemporal self-attention component Φ, and a decoder D.

[0023] In this invention, preferably, in step S2, the decoder in the first module is removed, and the feature encoding module E and the spatiotemporal self-attention component Φ are retained to form the feature extraction basic module ψ. The feature extraction basic module ψ is connected to the expression classification component to predict the expression of the input face video T-frame image and to constrain it using the expression category cross-entropy loss function. Then, the feature extraction basic module ψ is connected to the spatial self-attention component to process the input face video T-frame image to obtain feature encoding f1 and feature encoding f2, and then the cosine similarity loss function is used to constrain the feature similarity.

[0024] In this invention, preferably, in step S3, the multi-target supervision component adopts a fully connected network of multi-classification components.

[0025] In this invention, preferably, in step S3, the face authentication model takes a real or fake face video and a fake method label as input, and outputs a fake label q. i The objective function is expressed as:

[0026]

[0027] In the formula, i represents the number in the C categories, p i This label indicates the method of authentically forging the input video.

[0028] Compared with the prior art, the beneficial effects of the present invention are:

[0029] The deformable face authentication network of this invention fully considers the interference of semantic feature changes at the same position in fake face videos on forgery clues. It constrains the accurate learning of temporal inconsistencies between related semantic features in different spatial positions between frames by using prior guidance of temporal displacement of facial key points and a rigid offset objective function. The spatiotemporally consistent face authentication model construction method is based on the deformable face authentication network and the temporal self-attention feature mechanism, which can alleviate the interference of semantic feature shift between face video frames on temporal forgery feature extraction, thereby effectively improving the generalization ability of the authentication model. By constructing a spatiotemporally consistent face authentication model, the accurate separation and representation of spatiotemporal features of real faces and fake faces are achieved, thereby effectively improving the robustness of the authentication model to the authentication of unknown fake faces. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the deformable face authentication network process described in this invention.

[0031] Figure 2This is a schematic diagram illustrating the model construction process of a spatiotemporally consistent face authentication model construction method described in the invention.

[0032] Figure 3 This is a flowchart of the face video preprocessing process used as input to the face authentication model described in this invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0035] Please also see Figures 1 to 3 A preferred embodiment of the present invention provides a deformable face authentication network that fully considers the interference of semantic feature changes at the same position in fake face videos on fake clues. It uses prior guidance of temporal displacement of face key points and a rigid offset characteristic objective function to constrain the accurate learning of temporal inconsistencies between related semantic features in different spatial positions between frames. It mainly includes a backbone network and a deformable temporal self-attention network. The backbone network is used to extract face spatial features of the input image, and the deformable temporal self-attention network is used to process the face spatial features output by the backbone network to extract the temporal features of the face.

[0036] Specifically, in terms of spatial feature extraction, a backbone network based on the spatial self-attention mechanism of Transformer (self-attention transformation network) is adopted. In terms of temporal feature extraction, a deformable temporal self-attention network based on Transformer is adopted to get rid of the spatial location constraints of fake face videos and adaptively mine forgery inconsistency clues between similar semantic features in different frames, so as to mitigate the interference of face offset on temporal forgery features as much as possible.

[0037] Furthermore, before the model data is input into the face authentication model, the face video needs to be split into frames, and the faces need to be cropped and aligned. See the appendix for specific processing steps. Figure 3 .

[0038] In this embodiment, the backbone network mainly performs the following operations:

[0039] S01, Extract blocks from the input image and encode the blocks to obtain a block sequence;

[0040] S02, Block sequence features for processing block sequences.

[0041] In this embodiment, the deformable temporal self-attention network mainly performs the following operations:

[0042] S11, Input block sequence features, calculate the similarity between all blocks by dot product, and obtain the set of blocks with the highest similarity for each block in other positions in the current frame and in other frames;

[0043] S12, calculate the cross-attention between blocks and block sets to obtain the updated block spatiotemporal feature encoding, i.e. the temporal features of the face.

[0044] In this embodiment, the deformable temporal self-attention network calculates the key point offset between each frame using the known facial key point coordinates, and uses this as a reference value to constrain the accuracy of the prediction of high similarity blocks of key points between frames. At the same time, the positional offset between the block where the key point is located and the most similar block in other frames can be calculated by using other positions in the current frame and the block number with the highest similarity to the block in other frames.

[0045] Specifically, extract 16*16 blocks from the input video T frames and encode these blocks to obtain a block sequence F∈R. T×L×C (L is the block sequence length, C is the block encoding dimension), and then the block sequence is input into the spatial self-attention layer of the backbone network to obtain the block sequence features F. s ∈R T×L×C Deformable temporal self-attention networks use block sequence features F s As input, first calculate the similarity S between all blocks using dot product, and then obtain the similarity S between each block. The set of blocks with the highest similarity in other positions in the current frame and in other frames. (t represents the frame number, l represents the block number), and then by calculating and The updated block spatiotemporal feature encoding is obtained through cross-attention. The deformable temporal self-attention mechanism used in the deformable temporal self-attention network is expressed in the following formula:

[0046]

[0047] V,M = topk(softmax(S))

[0048]

[0049] in, For F s The transpose of M, where M is the position of other positions in the current frame and other frames. The block number with the highest similarity is obtained from this.

[0050] Furthermore, the learning of offsets in a deformable temporal self-attention network is guided by the inter-frame facial keypoint coordinate offsets. Taking advantage of the unidirectional nature of local facial offsets, an effective motion offset objective function is designed to constrain the learning of semantically related feature points between frames. The keypoint offsets between each frame are calculated using known facial keypoint coordinates. This value is used as a reference to constrain the accuracy of predicting high-similarity blocks of keypoints between frames. At the same time, M can be used to calculate the positional offset between the block containing the keypoint and the most similar blocks in other frames. The objective function is expressed as:

[0051]

[0052] Where K represents the number of facial landmarks, and k represents the index of the facial landmark. Furthermore, local facial offsets tend to be in the same direction; therefore, using a facial landmark as the center point, the offsets of its neighboring feature points are the same. The objective function is expressed as:

[0053]

[0054] in, This represents the average offset of all feature points within the face block centered on the k-th keypoint.

[0055] Another preferred embodiment of the present invention provides a method for constructing a spatiotemporally consistent face authentication model. Based on massive amounts of real face video data, and using a deformable face authentication network as a foundation, it combines self-supervised image reconstruction, contrastive learning, and supervised face video proxy tasks to model the spatiotemporal feature representation mechanism of real faces; and a spatiotemporal forgery feature extraction and multi-objective supervision mechanism for forged faces to alleviate the problem of uneven solution space of forged features as much as possible. The constructed face authentication model improves the practicality and robustness of authentication against both known and unknown forged faces. Specifically, it includes the following steps:

[0056] S1, Input T frames of the masked face video, which are processed by the first model to obtain the reconstructed face video.

[0057] S2, the first model is improved to obtain the second model, which uses spatiotemporal feature extraction and enhancement of the input face video T-frame images;

[0058] S3, based on the second model, incorporates a multi-target supervision component to form a face authentication model, which is used to identify whether the input face video is real or fake.

[0059] In this embodiment, the first model utilizes real face images and, drawing on the masking and training strategies of MAE, learns the spatial feature distribution of real faces. Specifically, the first model includes a feature encoding module E, a spatiotemporal self-attention component Φ, and a decoder D. The feature encoding module E and the spatiotemporal self-attention component Φ employ a deformable face authentication network structure. The input is a masked face video of T frames, which, after passing through the feature encoding module E, the spatiotemporal self-attention component Φ, and the decoder D, yields the reconstructed face video. In order to make Similar to the real face video V, the objective function is expressed as:

[0060]

[0061] In this embodiment, in step S2, the decoder in the first module is removed, and the feature encoding module E and the spatiotemporal self-attention component Φ are retained to form the feature extraction basic module Ψ. The feature extraction basic module Ψ is connected to the expression classification component to predict the expression of the input face video T-frame image and to constrain it with the expression category cross-entropy loss function. Then, the feature extraction basic module Ψ is connected to the spatial self-attention component to process the input face video T-frame image to obtain feature encoding f1 and feature encoding f2, and then the cosine similarity loss function is used to constrain the feature similarity.

[0062] Specifically, for the facial expression recognition task, the feature extraction module Ψ is used in conjunction with the facial expression classification component for processing. For the video frame interpolation task, the feature extraction module Ψ is used in conjunction with a fully connected network to predict the feature encoding f1 of the T-frame images. The feature encoding module E and the spatial self-attention component of the fully connected network are then used to process the feature encoding f2 of the T-frame images. Finally, the cosine similarity loss function is used to constrain the feature similarity, as expressed by the formula:

[0063]

[0064] In this embodiment, in step S3, the face authentication model is obtained by connecting the feature extraction module Ψ obtained in steps S1 and S2 to a multi-objective supervision component. The multi-objective supervision component adopts a fully connected network of multi-classification components. The face authentication model takes real or fake face videos and fake method labels as inputs and outputs a fake label Q. i The objective function is expressed as:

[0065]

[0066] In the formula, i represents the number in the C categories, pi This label indicates the method of authentically forging the input video.

[0067] The above description is a detailed description of the preferred embodiments of the present invention. However, the embodiments are not intended to limit the scope of the patent application of the present invention. All equivalent changes or modifications made under the technical spirit of the present invention should fall within the patent scope covered by the present invention.

Claims

1. A deformable face authentication network, characterized in that, It includes a backbone network and a deformable temporal self-attention network, wherein the backbone network is used to extract the face spatial features of the input image, and the deformable temporal self-attention network is used to process the face spatial features output by the backbone network to extract the temporal features of the face. The deformable temporal self-attention network calculates the key point offset between each frame using the known facial key point coordinates, and uses this as a reference value to constrain the accuracy of the prediction of high similarity blocks between key points. At the same time, it can calculate the positional offset between the block where the key point is located and the most similar block in other frames by using other positions in the current frame and the block number with the highest similarity to the block in other frames. The backbone network mainly performs the following operations: S01, Extract blocks from the input image and encode the blocks to obtain a block sequence; S02, Block sequence characteristics for processing block sequences; The deformable temporal self-attention network mainly performs the following operations: S11, Input block sequence features, calculate the similarity between all blocks by dot product, and obtain the set of blocks with the highest similarity for each block in other positions in the current frame and in other frames; S12, calculate the cross-attention between blocks and block sets to obtain the updated block spatiotemporal feature encoding, i.e. the temporal features of the face.

2. The deformable face authentication network according to claim 1, characterized in that, Both the backbone network and the deformable temporal self-attention network are based on the Transformer model.

3. A spatiotemporally consistent face authentication model construction method, employing the deformable face authentication network described in any one of claims 1-2, characterized in that, Includes the following steps: S1, input the masked face video T-frame image, and obtain the reconstructed face video through the first model; S2, the first model is improved to obtain the second model, which is enhanced by extracting and strengthening the spatiotemporal features of the input face video T-frame images; S3, based on the second model, incorporates a multi-target supervision component to form a face authentication model, used to identify the authenticity of input face videos.

4. The method for constructing a spatiotemporally consistent face authentication model according to claim 3, characterized in that, The first model includes a feature encoding module. Spatiotemporal self-attention component and decoder .

5. The method for constructing a spatiotemporally consistent face authentication model according to claim 4, characterized in that, In step S2, the decoder in the first module is removed, while the feature encoding module is retained. Spatiotemporal self-attention component Composition of basic feature extraction modules The feature extraction basic module The system integrates an expression classification component to predict expressions from T frames of input facial video images, using an expression category cross-entropy loss function for constraint. Then, the feature extraction module... The spatial self-attention component processes the input T-frame images of face video to obtain feature encoding. and feature encoding Furthermore, a cosine similarity loss function is used to constrain feature similarity.

6. The method for constructing a spatiotemporally consistent face authentication model according to claim 5, characterized in that, In step S3, the multi-target supervision component adopts a fully connected network of multi-classification components.

7. The method for constructing a spatiotemporally consistent face authentication model according to claim 6, characterized in that, In step S3, the face authentication model takes a real or fake face video and a fake method label as input, and outputs a fake label. The objective function is expressed as: In the formula, i express C Numbered in each category This label indicates the method of authentically forging the input video.

Images