An optical flow guided spatial alignment face spoofing detection method and system

Abstract

The application discloses an image processing technical field and relates to a face forgery detection method and system guided by optical flow and spatial alignment. The method comprises the following steps: a pre-processing and optical flow extraction module is used to pre-process three continuous frames of face images to be detected, so as to obtain three continuous frames of pre-processed face images, and a bidirectional optical flow field is calculated based on the three continuous frames of pre-processed face images; a time sequence motion verification module is used to verify the motion smoothness of the bidirectional optical flow field, generate a continuous motion error representing a forgery risk, and generate a time sequence basic feature; a light flow guided spatial alignment module is used to generate a spatial basic feature according to the three continuous frames of pre-processed face images, the bidirectional optical flow field and the continuous motion error; a progressive space-time fusion module is used to perform multi-scale hierarchical fusion on the time sequence basic feature and the spatial basic feature, so as to obtain a space-time joint forgery feature; and a classifier is used to generate a detection result according to the space-time joint forgery feature.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for detecting face forgery using optical flow-guided spatial alignment. Background Technology

[0002] With the rapid development of generative artificial intelligence technology, the methods and forms of facial forgery are constantly evolving. The emergence of new generative technologies such as generative adversarial networks, diffusion models, and neural rendering has enabled a large number of fake facial images and videos to achieve a level of visual realism that is indistinguishable from reality. Especially in video scenarios, high-fidelity deepfake content can approach realism in terms of single-frame image quality, texture detail, and facial expressions. If such technologies are maliciously exploited, they will directly threaten the authenticity of facial identity authentication and online video content, leading to security risks such as identity theft, misleading public opinion, and telecommunications fraud.

[0003] Therefore, face forgery detection technology has emerged to accurately determine the authenticity of face images or videos. Existing methods typically uncover forgery clues from the perspectives of single-frame texture anomalies, frequency domain statistical characteristics, local artifacts, or optical flow motion features, achieving certain performance improvements on specific datasets and forgery types. However, most of these methods still rely on low-quality output traces left by the forgery generator itself. With the continuous development of high-fidelity face forgery technology, these traces are becoming increasingly weak, causing existing detection methods to experience a significant performance decline when facing new forgery techniques, making stable transfer difficult.

[0004] Furthermore, most existing detection methods focus only on modeling independent spatial or temporal features, often ignoring the inherent strong correlation between inter-frame motion and spatial tampering traces of faces. This makes it difficult to fully explore the deep correlation defects of forged samples in motion and spatial dimensions, resulting in limited generalization ability when facing new forgery methods.

[0005] In fact, the physical relationship between inter-frame motion and spatial texture structure of faces can be characterized by optical flow: in real-world scenarios, the spatial pixel displacement of consecutive face frames and the motion trend represented by optical flow have an inherent consistency, and the face structure between frames exhibits strict motion continuity; however, due to frame-by-frame generation or local tampering, forged face videos often create imperceptible gaps between the consistency of inter-frame motion and the spatial texture structure. Therefore, compared to simply mining forgery traces from a single dimension, how to fully explore the correlation between the inter-frame motion information carried by optical flow and spatial structure, and thus characterize the essential defects of forged samples, has become a key issue in improving the generalization ability of face forgery detection. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a face forgery detection method and system with optical flow-guided spatial alignment. The aim is to establish the correlation between inter-frame motion information and spatial texture features by using the motion consistency between consecutive face frames as a constraint and optical flow-guided spatial alignment. Based on this, the forgery traces of the input sample in the motion and spatial dimensions are estimated, thereby achieving face forgery detection with high generalization.

[0007] To solve the above-mentioned technical problems, the present invention is implemented using the following technical solution:

[0008] In a first aspect, the present invention provides a face forgery detection method with optical flow-guided spatial alignment, comprising:

[0009] Three consecutive frames of face images to be detected are input into a trained detection model to perform forgery detection and obtain the detection results.

[0010] The forgery detection includes:

[0011] The three consecutive frames of face images to be detected are preprocessed using a preprocessing and optical flow extraction module to obtain three consecutive frames of face images after preprocessing, and a bidirectional optical flow field is calculated based on the three consecutive frames of face images after preprocessing.

[0012] The motion smoothness of the bidirectional optical flow field is verified using a time-series motion verification module, generating continuous motion errors that characterize the risk of forgery, and generating time-series basic features.

[0013] Based on the preprocessed three consecutive frames of face images, bidirectional optical flow field and continuous motion error, spatial basic features are generated using the optical flow-guided spatial alignment module.

[0014] The temporal and spatial basic features are fused at multiple scales using a progressive spatiotemporal fusion module to obtain spatiotemporal joint forgery features.

[0015] Based on the spatiotemporal joint forgery features, a classifier is used to generate detection results.

[0016] Optionally, the preprocessing and optical flow extraction module includes a preprocessing unit and a RAFT-Lite model;

[0017] The preprocessing unit is used to perform face detection, key point localization, rigid alignment, cropping and scaling, and pixel normalization on the three consecutive frames of face images to be detected, so as to obtain the three consecutive frames of face images after preprocessing.

[0018] The RAFT-Lite model is used to calculate the bidirectional optical flow field based on the three consecutive preprocessed face images;

[0019] The bidirectional optical flow field includes a first forward optical flow field, a second forward optical flow field, a first reverse optical flow field, and a second reverse optical flow field.

[0020] Optionally, the continuous motion error is obtained by the following formula:

[0021] ,

[0022] ,

[0023] ,

[0024] in, Indicates the first Frame to the The per-pixel motion velocity field of the frame. This represents the first forward optical flow field of three consecutive frames of face images after preprocessing. Indicates the horizontal displacement of a pixel. Indicates the vertical displacement of a pixel. Represents the first forward optical flow field Pixel horizontal displacement, Represents the first forward optical flow field The vertical displacement of pixels, Indicates the first Frame to the The per-pixel motion velocity field of the frame. This represents the second forward optical flow field of three consecutive frames of face images after preprocessing. Represents the second forward optical flow field Pixel horizontal displacement, Represents the second forward optical flow field The vertical displacement of pixels, Indicates continuous motion error;

[0025] The fundamental time-series features are obtained through the following formula:

[0026]

[0027] in, Representing the basic characteristics of time series, This represents the first encoder, which consists of a set of stacked convolutional layers.

[0028] Optionally, the optical flow-guided spatial alignment module includes a coordinate alignment unit, a feature stitching unit, an attention weighting unit, and a second encoder;

[0029] The coordinate alignment unit is used to align the first and third frames of the three consecutive preprocessed face images to the reference coordinate system of the middle frame by multiplexing the Warp operation according to the bidirectional optical flow field, so as to obtain the aligned three face images.

[0030] The feature stitching unit is used to stitch the three aligned face images along the channel dimension and output a multi-frame joint spatial feature map.

[0031] The attention weighting unit is used to generate a fake region attention mask based on the continuous motion error, and to perform channel-by-channel weighting on the joint spatial feature map of multiple frames based on the fake region attention mask to obtain a weighted and enhanced feature map.

[0032] The second encoder is used to extract features from the weighted and enhanced feature map and output spatial basic features; wherein, the second encoder is composed of a set of stacked convolutional layers.

[0033] Optionally, the aligned three frames of face images are obtained using the following formula:

[0034] ,

[0035] ,

[0036] ,

[0037] in, This represents the first frame of the three aligned face images. This represents the third frame of the aligned three-frame face image. This represents the second frame of the three aligned face images. This represents the first frame of three consecutive preprocessed face images. This represents the third frame of a three-frame face image after preprocessing. This represents the second frame of a three-frame face image after preprocessing. This represents the first forward optical flow field of three consecutive frames of face images after preprocessing. This represents the second forward optical flow field of three consecutive frames of face images after preprocessing.

[0038] The multi-frame joint spatial feature map is obtained by the following formula:

[0039] ,

[0040] in, Represents a joint spatial feature map of multiple frames. Indicates a splicing operation;

[0041] The weighted enhanced feature map is obtained by the following formula:

[0042] ,

[0043] ,

[0044] in, This represents the weighted enhanced feature map. This indicates an element-wise multiplication operation. This indicates a forged regional attention mask. This represents the Sigmoid activation function. Indicates continuous motion error;

[0045] The basic spatial features are obtained through the following formula:

[0046] ,

[0047] in, Representing the basic characteristics of space, This indicates the second encoder.

[0048] Optionally, the progressive spatiotemporal fusion module includes a pixel-level fusion unit and a channel-level fusion unit;

[0049] The pixel-level fusion unit is used to perform pixel-level adaptive weighted addition of the temporal basic features and spatial basic features to obtain pixel-level fused features;

[0050] The channel-level fusion unit uses a convolutional module to map the pixel-level fusion features to a high-dimensional feature space to obtain high-dimensional pixel-level fusion features. Then, a channel weight vector of the high-dimensional pixel-level fusion features is generated through a channel attention module. Finally, the high-dimensional pixel-level fusion features are enhanced channel by channel weight based on the channel weight vector to obtain spatiotemporal joint forgery features. The convolutional module includes two 3×3 convolutional layers, a batch normalization layer, and a ReLU activation function.

[0051] Optionally, the pixel-level fusion feature is obtained by the following formula:

[0052] ,

[0053] ,

[0054] in, Represents pixel-level fusion features. This represents a pixel-level learnable adaptive fusion weight tensor. Representing the basic characteristics of time series, The symbol ⊙ represents the fundamental characteristics of space, and ⊙ represents the element-wise multiplication operation. This represents a 1×1 convolution operation. This indicates a splicing operation. This represents the Sigmoid activation function;

[0055] The high-dimensional pixel-level fusion feature is obtained through the following formula:

[0056] ,

[0057] in, Represents high-dimensional pixel-level fusion features. Indicates the operation of the convolution module;

[0058] The channel weight vector of the high-dimensional pixel-level fusion feature is obtained by the following formula:

[0059] ,

[0060] ,

[0061] ,

[0062] in, The channel weight vector represents the high-dimensional pixel-level fusion features. This represents a two-layer fully connected network with shared weights. This represents the mean descriptor. This represents a global average pooling operation along the channel dimension. Represents the maximum value descriptor. This represents the global max pooling operation along the channel dimension;

[0063] The spatiotemporal joint forgery feature is obtained through the following formula:

[0064] ,

[0065] in, ⊙ represents spatiotemporal joint forgery features, and ⊙ represents element-wise multiplication operation.

[0066] Optionally, the classifier includes a fully connected layer and a Sigmoid activation function.

[0067] Optionally, the training process of the detection model includes:

[0068] The pre-built detection model is iteratively updated and trained based on the pre-built training set. When the preset training stopping condition is reached, the model parameters that minimize the cross-entropy loss are loaded into the pre-built detection model to obtain the trained detection model. Each sample in the training set is a real or three consecutive frames of face images generated by existing forgery technology. The real label of each sample is 0 or 1, where 0 represents a real face and 1 represents a forged face.

[0069] A single iteration of training includes:

[0070] The detection model is used to generate detection results for each sample in the training set;

[0071] Based on the detection results and the true label of each sample, calculate the cross-entropy loss;

[0072] Based on the cross-entropy loss, the model parameters of the detection model are updated using the gradient descent method.

[0073] Secondly, the present invention provides a face forgery detection system with optical flow-guided spatial alignment, comprising:

[0074] The detection module is used to input three consecutive frames of face images to be detected as genuine or fake into a trained detection model to perform forgery detection and obtain the detection results.

[0075] The preprocessing and optical flow extraction module is used to preprocess the three consecutive frames of face images to be detected, obtain the three consecutive frames of face images after preprocessing, and calculate the bidirectional optical flow field based on the three consecutive frames of face images after preprocessing.

[0076] The temporal motion verification module is used to verify the motion smoothness of the bidirectional optical flow field, generate continuous motion errors that characterize the risk of forgery, and generate temporal basic features.

[0077] The optical flow-guided spatial alignment module is used to generate basic spatial features based on the preprocessed three consecutive frames of face images, the bidirectional optical flow field, and continuous motion errors.

[0078] A progressive spatiotemporal fusion module is used to perform multi-scale hierarchical fusion of the temporal basic features and spatial basic features to obtain spatiotemporal joint forgery features;

[0079] The classifier module is used to generate detection results based on the spatiotemporal joint forgery features.

[0080] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0081] 1. The preprocessing and optical flow extraction module calculates the bidirectional optical flow field using three consecutive frames of face images. The temporal motion verification module filters natural motion noise through motion smoothness verification, generating continuous motion errors and temporal basic features that characterize the risk of forgery. The optical flow-guided spatial alignment module completes the system unification of the spatial coordinates of the three frames of faces using the bidirectional optical flow field. Combined with continuous motion errors, it achieves attention weighting enhancement of the forgery region, eliminates the interference of natural motion on spatial artifact detection, and realizes full-process guidance of temporal motion information for spatial feature extraction. The progressive spatiotemporal fusion module uses a multi-scale hierarchical fusion mechanism to realize cross-branch deep information interaction between temporal features and spatial features, achieving deep coupling of spatiotemporal branches.

[0082] 2. It can effectively detect the essential disconnect between motion consistency and spatial tampering traces in high-fidelity deepfake videos, improve the accuracy and generalization ability of face forgery detection in unknown forgery types and cross-dataset scenarios, and can be widely used in video forgery detection, judicial video evidence collection and financial identity verification scenarios. Attached Figure Description

[0083] Figure 1 This is a flowchart of a face forgery detection method with optical flow-guided spatial alignment provided according to an embodiment of the present invention;

[0084] Figure 2 This is a data processing flowchart of the timing motion verification module provided according to an embodiment of the present invention;

[0085] Figure 3 This is a data processing flowchart of the optical flow-guided spatial alignment module provided in an embodiment of the present invention;

[0086] Figure 4 This is a data processing flowchart for a progressive spatiotemporal fusion module provided according to an embodiment of the present invention. Detailed Implementation

[0087] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations thereof. In the absence of conflict, the embodiments and technical features in the embodiments can be combined with each other.

[0088] It should be noted that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0089] Example 1

[0090] This invention discloses a face forgery detection method with optical flow-guided spatial alignment, referencing... Figure 1 As shown, the specific steps include the following:

[0091] Three consecutive frames of face images to be detected are input into a trained detection model to perform forgery detection and obtain the detection results.

[0092] The forgery detection includes:

[0093] S1, the three consecutive frames of face images to be detected are preprocessed using the preprocessing and optical flow extraction module to obtain the three consecutive frames of face images after preprocessing, and the bidirectional optical flow field is calculated based on the three consecutive frames of face images after preprocessing.

[0094] S2, the motion smoothness of the bidirectional optical flow field is verified by the temporal motion verification module, a continuous motion error characterizing the risk of forgery is generated, and a temporal basic feature is generated.

[0095] S3, based on the preprocessed three consecutive frames of face images, bidirectional optical flow field and continuous motion error, generate spatial basic features using the optical flow-guided spatial alignment module;

[0096] S4, using a progressive spatiotemporal fusion module to perform multi-scale hierarchical fusion of the temporal basic features and spatial basic features to obtain spatiotemporal joint forgery features;

[0097] S5. Based on the spatiotemporal joint forgery features, a classifier is used to generate detection results.

[0098] In step S1, the preprocessing and optical flow extraction module includes a preprocessing unit and a RAFT-Lite model;

[0099] The preprocessing unit is used to perform face detection, key point localization, rigid alignment, cropping and scaling, and pixel normalization on the three consecutive frames of face images to be detected, so as to obtain the three consecutive frames of face images after preprocessing.

[0100] The RAFT-Lite model is used to calculate a bidirectional optical flow field based on the three consecutive preprocessed face images; the bidirectional optical flow field includes a first forward optical flow field, a second forward optical flow field, a first reverse optical flow field, and a second reverse optical flow field.

[0101] The bidirectional optical flow fields are all dual-channel tensors. Channel 0 For pixel horizontal displacement, channel 1 The vertical displacement of a pixel is expressed as follows:

[0102]

[0103] in, For the real number field, For batch size, Indicates the height of the face image. This indicates the width of the face image.

[0104] In step S2, the temporal motion verification module, based on the bidirectional optical flow field and pixel motion information of the face image, generates continuous motion errors characterizing the risk of forgery through motion smoothness verification. Its data processing flow is as follows: Figure 2 As shown, the continuous motion error is obtained by the following formula:

[0105] ,

[0106] ,

[0107] ,

[0108] in, Indicates the first Frame to the The per-pixel motion velocity field of the frame. This represents the first forward optical flow field of three consecutive frames of face images after preprocessing. Indicates the horizontal displacement of a pixel. Indicates the vertical displacement of a pixel. Represents the first forward optical flow field Pixel horizontal displacement, Represents the first forward optical flow field The vertical displacement of pixels, Indicates the first Frame to the The per-pixel motion velocity field of the frame. This represents the second forward optical flow field of three consecutive frames of face images after preprocessing. Represents the second forward optical flow field Pixel horizontal displacement, Represents the second forward optical flow field The vertical displacement of pixels, Indicates continuous motion error;

[0109] The fundamental time-series features are obtained through the following formula:

[0110]

[0111] in, Representing the basic characteristics of time series, This represents the first encoder, consisting of a set of stacked convolutional layers, used to map continuous motion errors from a single channel into a high-dimensional temporal feature space.

[0112] In step S3, refer to Figure 3 As shown, the optical flow-guided spatial alignment module includes a coordinate alignment unit, a feature stitching unit, an attention weighting unit, and a second encoder.

[0113] The coordinate alignment unit is used to align the first and third frames of the preprocessed three consecutive face images to the reference coordinate system of the middle frame by multiplexing the Warp operation according to the bidirectional optical flow field, so that the facial features and pixel positions of the three frames of the face are completely one-to-one, eliminating the interference of natural motion on spatial artifact detection, and obtaining the aligned three-frame face images:

[0114] ,

[0115] ,

[0116] ,

[0117] in, This represents the first frame of the three aligned face images. This represents the third frame of the aligned three-frame face image. This represents the second frame of the three aligned face images. This represents the first frame of three consecutive preprocessed face images. This represents the third frame of a three-frame face image after preprocessing. This represents the second frame of a three-frame face image after preprocessing. This represents the first forward optical flow field of three consecutive frames of face images after preprocessing. This represents the second forward optical flow field of three consecutive frames of face images after preprocessing.

[0118] The feature stitching unit is used to stitch the aligned three frames of face images along the channel dimension, enabling the spatial branch to simultaneously capture single-frame static artifacts and inter-frame tampering differences, and output a multi-frame joint spatial feature map:

[0119] ,

[0120] in, Represents a joint spatial feature map of multiple frames. This indicates a splicing operation.

[0121] The attention weighting unit is used to generate a fake region attention mask based on the continuous motion error, so that the spatial branch automatically focuses on the fake high-risk region marked by optical flow, suppresses invalid noise in the background and normal face regions, and performs channel-by-channel weighting on the joint spatial feature map of multiple frames based on the fake region attention mask to obtain a weighted and enhanced feature map:

[0122] ,

[0123] ,

[0124] in, This represents the weighted enhanced feature map. This indicates an element-wise multiplication operation. This indicates a forged regional attention mask. This represents the Sigmoid activation function. This represents the error in continuous motion.

[0125] The second encoder is used to extract features from the weighted and enhanced feature map, and outputs the basic spatial features:

[0126] ,

[0127] in, Representing the basic characteristics of space, This represents the second encoder, whose network structure is the same as the first encoder, but the two are independent of each other and their weights are not shared. The second encoder is used to map the 9-channel weighted feature map into a high-dimensional space.

[0128] In step S4, refer to Figure 4 As shown, the progressive spatiotemporal fusion module includes pixel-level fusion units and channel-level fusion units.

[0129] The pixel-level fusion unit is used to perform pixel-level adaptive weighted addition of the temporal and spatial basic features, preserving fine-grained forgery traces at the original spatial resolution of the two branches, avoiding the loss of boundary and texture artifacts caused by downsampling, and obtaining pixel-level fused features:

[0130] ,

[0131] ,

[0132] in, Represents pixel-level fusion features. This represents a pixel-level learnable adaptive fusion weight tensor. Representing the basic characteristics of time series, The symbol ⊙ represents the fundamental characteristics of space, and ⊙ represents the element-wise multiplication operation. This represents a 1×1 convolution operation. This indicates a splicing operation. This represents the Sigmoid activation function; the value of each pixel in the pixel-level fusion feature is adaptively determined by the forgery credibility of the two branch features at that position: when the continuous motion error is large, Approaching 1, shallow fusion features are dominated by temporal motion anomalies; when spatial texture artifacts are significant, Approaching 0, shallow fusion features are dominated by evidence of spatial tampering.

[0133] The channel-level fusion unit, based on pixel-level fusion features, models the differences in forgery responses between different channels through a channel attention mechanism, adaptively amplifying high-discrimination channels for forgery and suppressing redundant background channels to obtain spatiotemporal joint forgery features. Specifically, the convolution module maps the pixel-level fusion features to a high-dimensional feature space to obtain high-dimensional pixel-level fusion features, and the channel attention module generates channel weight vectors for the high-dimensional pixel-level fusion features. Finally, based on the channel weight vectors, the high-dimensional pixel-level fusion features are enhanced channel by channel to obtain spatiotemporal joint forgery features. The convolution module includes two 3×3 convolution layers, a batch normalization layer, and a ReLU activation function.

[0134] The high-dimensional pixel-level fusion feature is obtained through the following formula:

[0135] ,

[0136] in, Represents high-dimensional pixel-level fusion features. Indicates the operation of the convolution module;

[0137] The channel weight vector of the high-dimensional pixel-level fusion feature is obtained by the following formula:

[0138] ,

[0139] ,

[0140] ,

[0141] in, The channel weight vector represents the high-dimensional pixel-level fusion features. This represents a two-layer fully connected network with shared weights. This represents the mean descriptor. This represents a global average pooling operation along the channel dimension. Represents the maximum value descriptor. This represents the global max pooling operation along the channel dimension;

[0142] The spatiotemporal joint forgery feature is obtained through the following formula:

[0143] ,

[0144] in, ⊙ represents spatiotemporal joint forgery features, and ⊙ represents element-wise multiplication operation.

[0145] The two-level fusion mechanism of the progressive spatiotemporal fusion module has sequential dependence. Pixel-level fusion preserves pixel-level details and completes the initial coupling of temporal and spatial branches. Channel-level fusion enhances channel-level discriminative power on this basis. The final spatiotemporal joint forgery feature has both local details and channel discriminative power.

[0146] In step S5, the classifier includes a fully connected layer and a sigmoid activation function; the input size of the fully connected layer is the size of the feature vector after global average pooling of the spatiotemporal joint forgery features; the output size of the fully connected layer is 1; the sigmoid activation function normalizes the output of the fully connected layer to a forgery probability P_fake in the interval [0, 1], and P_fake is judged as forgery when P_fake≥0.5, otherwise it is judged as real.

[0147] The training process of the detection model includes:

[0148] The pre-built detection model is iteratively updated and trained based on the pre-built training set. When the preset training stopping condition is reached, the model parameters that minimize the cross-entropy loss are loaded into the pre-built detection model to obtain the trained detection model. Each sample in the training set is a real or three consecutive frames of face images generated by existing forgery technology. The real label of each sample is 0 or 1, where 0 represents a real face and 1 represents a forged face.

[0149] A single iteration of training includes:

[0150] The detection model is used to generate detection results for each sample in the training set;

[0151] Based on the detection results and the true label of each sample, calculate the cross-entropy loss;

[0152] Based on the cross-entropy loss, the model parameters of the detection model are updated using the gradient descent method.

[0153] The training set in this embodiment includes a publicly available dataset that can be downloaded directly and a privately constructed dataset; the publicly available dataset includes FaceForensics++ (FF++) and DeepFake Detection Challenge (DFDC).

[0154] The construction of a private dataset includes the following steps:

[0155] The system obtains real face videos and face videos generated using existing forgery techniques from video websites; it extracts frames from the videos at a fixed frame rate, and uses the Dlib tool to perform face detection and locate 68 facial key points in each frame; in this embodiment, the fixed frame rate is set to 25 fps.

[0156] Based on 68 facial key points, affine transformation is used to rigidly align each frame of face to the standard coordinate system, ensuring cross-frame consistency of face structure between consecutive frames.

[0157] After setting fixed margins for the aligned face area, crop it.

[0158] The cropped face image is scaled to a fixed pixel size to obtain a preprocessed face frame sequence;

[0159] Three consecutive frames are extracted from the face frame sequence using a sliding window method as a minimum detection time window to form the training samples for the model.

[0160] Each training sample corresponds to a real label y∈{0, 1}, where y=0 indicates that the sample comes from a real face video, and y=1 indicates that the sample comes from a fake face video.

[0161] The cross-entropy loss is obtained using the following formula:

[0162]

[0163] in, This represents the true label of the input training sample. This indicates the detection result of the detection model on the training sample. express and The cross-entropy loss.

[0164] In summary, the optical flow-guided spatial alignment face forgery detection method proposed in this embodiment firstly preprocesses three consecutive face video frames to obtain standardized aligned three-frame face images and calculates the bidirectional optical flow field. Secondly, the temporal motion verification module performs motion smoothness verification based on the bidirectional optical flow field to obtain continuous motion errors and temporal basic features characterizing forgery risk. Then, the optical flow-guided spatial alignment module aligns the three face frames to a unified coordinate system based on the bidirectional optical flow field and generates a forgery region attention mask based on the continuous motion errors, extracting the weighted enhanced features to obtain spatial basic features. Furthermore, the progressive spatiotemporal fusion module performs pixel-level and channel-level multi-scale fusion of the temporal and spatial basic features to obtain spatiotemporal joint forgery features. Finally, the classifier outputs the authenticity judgment result of the face video based on the spatiotemporal joint forgery features. This method can effectively detect the essential disconnect between motion consistency and spatial tampering traces in high-fidelity deepfake videos and can be widely applied in scenarios such as video forgery detection, judicial video evidence collection, and financial identity verification.

[0165] Example 2

[0166] Based on the same inventive concept as Embodiment 1, this embodiment of the invention discloses a face forgery detection system with optical flow-guided spatial alignment, comprising:

[0167] The detection module is used to input three consecutive frames of face images to be detected as genuine or fake into a trained detection model to perform forgery detection and obtain the detection results.

[0168] The preprocessing and optical flow extraction module is used to preprocess the three consecutive frames of face images to be detected, obtain the three consecutive frames of face images after preprocessing, and calculate the bidirectional optical flow field based on the three consecutive frames of face images after preprocessing.

[0169] The temporal motion verification module is used to verify the motion smoothness of the bidirectional optical flow field, generate continuous motion errors that characterize the risk of forgery, and generate temporal basic features.

[0170] The optical flow-guided spatial alignment module is used to generate basic spatial features based on the preprocessed three consecutive frames of face images, the bidirectional optical flow field, and continuous motion errors.

[0171] A progressive spatiotemporal fusion module is used to perform multi-scale hierarchical fusion of the temporal basic features and spatial basic features to obtain spatiotemporal joint forgery features;

[0172] The classifier module is used to generate detection results based on the spatiotemporal joint forgery features.

[0173] The specific functions of each module described above are explained in the relevant content of the method in Embodiment 1, and will not be repeated here.

[0174] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0175] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0176] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0177] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0178] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for detecting face forgery using optical flow-guided spatial alignment, characterized in that, include: Three consecutive frames of face images to be detected are input into a trained detection model to perform forgery detection and obtain the detection results. The forgery detection includes: The three consecutive frames of face images to be detected are preprocessed using a preprocessing and optical flow extraction module to obtain three consecutive frames of face images after preprocessing, and a bidirectional optical flow field is calculated based on the three consecutive frames of face images after preprocessing. The motion smoothness of the bidirectional optical flow field is verified using a time-series motion verification module, generating continuous motion errors that characterize the risk of forgery, and generating time-series basic features. Based on the preprocessed three consecutive frames of face images, bidirectional optical flow field and continuous motion error, spatial basic features are generated using the optical flow-guided spatial alignment module. The temporal and spatial basic features are fused at multiple scales using a progressive spatiotemporal fusion module to obtain spatiotemporal joint forgery features. Based on the spatiotemporal joint forgery features, a classifier is used to generate detection results.

2. The face forgery detection method guided by optical flow spatial alignment according to claim 1, characterized in that, The preprocessing and optical flow extraction module includes a preprocessing unit and a RAFT-Lite model; The preprocessing unit is used to perform face detection, key point localization, rigid alignment, cropping and scaling, and pixel normalization on the three consecutive frames of face images to be detected, so as to obtain the three consecutive frames of face images after preprocessing. The RAFT-Lite model is used to calculate the bidirectional optical flow field based on the three consecutive preprocessed face images. The bidirectional optical flow field includes a first forward optical flow field, a second forward optical flow field, a first reverse optical flow field, and a second reverse optical flow field.

3. The face forgery detection method guided by optical flow spatial alignment according to claim 1, characterized in that, The continuous motion error is obtained by the following formula: ， ， ， in, Indicates the first Frame to the The per-pixel motion velocity field of the frame. This represents the first forward optical flow field of three consecutive frames of face images after preprocessing. Indicates the horizontal displacement of a pixel. Indicates the vertical displacement of a pixel. Represents the first forward optical flow field Pixel horizontal displacement, Represents the first forward optical flow field The vertical displacement of pixels, Indicates the first Frame to the The per-pixel motion velocity field of the frame. This represents the second forward optical flow field of three consecutive frames of face images after preprocessing. Represents the second forward optical flow field Pixel horizontal displacement, Represents the second forward optical flow field The vertical displacement of pixels, Indicates continuous motion error; The fundamental time-series features are obtained through the following formula: in, Representing the basic characteristics of time series, This represents the first encoder, which consists of a set of stacked convolutional layers.

4. The face forgery detection method guided by optical flow spatial alignment according to claim 1, characterized in that, The optical flow-guided spatial alignment module includes a coordinate alignment unit, a feature stitching unit, an attention weighting unit, and a second encoder. The coordinate alignment unit is used to align the first and third frames of the three consecutive preprocessed face images to the reference coordinate system of the middle frame by multiplexing the Warp operation according to the bidirectional optical flow field, so as to obtain the aligned three face images. The feature stitching unit is used to stitch the three aligned face images along the channel dimension and output a multi-frame joint spatial feature map. The attention weighting unit is used to generate a fake region attention mask based on the continuous motion error, and to perform channel-by-channel weighting on the joint spatial feature map of multiple frames based on the fake region attention mask to obtain a weighted and enhanced feature map. The second encoder is used to extract features from the weighted and enhanced feature map and output spatial basic features; wherein, the second encoder is composed of a set of stacked convolutional layers.

5. The face forgery detection method guided by optical flow spatial alignment according to claim 4, characterized in that, The aligned three frames of face images are obtained using the following formula: ， ， ， in, This represents the first frame of the three aligned face images. This represents the third frame of the aligned three-frame face image. This represents the second frame of the three aligned face images. This represents the first frame of three consecutive preprocessed face images. This represents the third frame of a three-frame face image after preprocessing. This represents the second frame of a three-frame face image after preprocessing. This represents the first forward optical flow field of three consecutive frames of face images after preprocessing. This represents the second forward optical flow field of three consecutive frames of face images after preprocessing. The multi-frame joint spatial feature map is obtained by the following formula: ， in, Represents a joint spatial feature map of multiple frames. Indicates a splicing operation; The weighted enhanced feature map is obtained by the following formula: ， ， in, This represents the weighted enhanced feature map. This indicates an element-wise multiplication operation. This indicates a forged regional attention mask. This represents the Sigmoid activation function. Indicates continuous motion error; The basic spatial features are obtained through the following formula: ， in, Representing the basic characteristics of space, This indicates the second encoder.

6. The face forgery detection method guided by optical flow spatial alignment according to claim 1, characterized in that, The progressive spatiotemporal fusion module includes pixel-level fusion units and channel-level fusion units; The pixel-level fusion unit is used to perform pixel-level adaptive weighted addition of the temporal basic features and spatial basic features to obtain pixel-level fused features; The channel-level fusion unit uses a convolutional module to map the pixel-level fusion features to a high-dimensional feature space to obtain high-dimensional pixel-level fusion features. Then, a channel weight vector of the high-dimensional pixel-level fusion features is generated through a channel attention module. Finally, the high-dimensional pixel-level fusion features are enhanced channel by channel weight based on the channel weight vector to obtain spatiotemporal joint forgery features. The convolutional module includes two 3×3 convolutional layers, a batch normalization layer, and a ReLU activation function.

7. The face forgery detection method guided by optical flow spatial alignment according to claim 6, characterized in that, The pixel-level fusion feature is obtained through the following formula: ， ， in, Indicates pixel-level fusion features, This represents a pixel-level learnable adaptive fusion weight tensor. Representing the basic characteristics of time series, The symbol ⊙ represents the fundamental characteristics of space, and ⊙ represents the element-wise multiplication operation. This represents a 1×1 convolution operation. This indicates a splicing operation. This represents the Sigmoid activation function; The high-dimensional pixel-level fusion feature is obtained through the following formula: ， in, Represents high-dimensional pixel-level fusion features. Indicates the operation of the convolution module; The channel weight vector of the high-dimensional pixel-level fusion feature is obtained by the following formula: ， ， ， in, The channel weight vector represents the high-dimensional pixel-level fusion features. This represents a two-layer fully connected network with shared weights. This represents the mean descriptor. This represents a global average pooling operation along the channel dimension. Represents the maximum value descriptor. This represents the global max pooling operation along the channel dimension; The spatiotemporal joint forgery feature is obtained through the following formula: ， in, ⊙ represents spatiotemporal joint forgery features, and ⊙ represents element-wise multiplication operation.

8. The face forgery detection method with optical flow-guided spatial alignment according to claim 1, characterized in that, The classifier consists of a fully connected layer and a Sigmoid activation function.

9. The face forgery detection method with optical flow-guided spatial alignment according to claim 1, characterized in that, The training process of the detection model includes: The pre-built detection model is iteratively updated and trained based on the pre-built training set. When the preset training stopping condition is reached, the model parameters that minimize the cross-entropy loss are loaded into the pre-built detection model to obtain the trained detection model. Each sample in the training set is a real or three consecutive frames of face images generated by existing forgery technology. The real label of each sample is 0 or 1, where 0 represents a real face and 1 represents a forged face. A single iteration of training includes: The detection model is used to generate detection results for each sample in the training set; Based on the detection results and the true label of each sample, calculate the cross-entropy loss; Based on the cross-entropy loss, the model parameters of the detection model are updated using the gradient descent method.

10. A face forgery detection system with optical flow-guided spatial alignment, characterized in that, include: The detection module is used to input three consecutive frames of face images to be detected as genuine or fake into a trained detection model to perform forgery detection and obtain the detection results. The preprocessing and optical flow extraction module is used to preprocess the three consecutive frames of face images to be detected, obtain the three consecutive frames of face images after preprocessing, and calculate the bidirectional optical flow field based on the three consecutive frames of face images after preprocessing. The temporal motion verification module is used to verify the motion smoothness of the bidirectional optical flow field, generate continuous motion errors that characterize the risk of forgery, and generate temporal basic features. The optical flow-guided spatial alignment module is used to generate basic spatial features based on the preprocessed three consecutive frames of face images, the bidirectional optical flow field, and continuous motion errors. A progressive spatiotemporal fusion module is used to perform multi-scale hierarchical fusion of the temporal basic features and spatial basic features to obtain spatiotemporal joint forgery features; The classifier module is used to generate detection results based on the spatiotemporal joint forgery features.

Images