A face-spoofing video detection method and system

Abstract

The application provides a face fake video detection method and system. The method comprises the following steps: 1) preprocessing the input video containing a face to obtain a sequence of frame-by-frame face image frames S; 2) using a pre-trained convolutional neural network to extract a visual feature map F of the face image; 3) cutting and expanding the feature map into a token vector sequence T; 4) preprocessing the token vector sequence, adding classification tokens (T t and T s ) in the time dimension and the space dimension respectively to obtain a token sequence I as the input of the transformer; 5) using a video transformer neural network to process the token sequence; 6) using a classification pooling layer to obtain tokens for classification, and using a prediction head to predict the classification result. Compared with the prior art, the application effectively utilizes the time sequence information existing in the video, has the advantages of strong robustness and good interpretability.

Description

Technical Field

[0001] This invention relates to the field of intelligent facial video analysis, and in particular to a method and system for detecting fake facial videos based on a spatiotemporal separation self-attention video transformer. Background Technology

[0002] Deep learning-based face spoofing, or Deepfake, refers to the use of deep models based on Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) to modify or replace target faces in videos, achieving the purpose of tampering or face swapping. Clearly, face spoofing technology poses a serious threat to our personal privacy and cybersecurity. However, existing research shows that the general public has poor ability to identify spoofed faces, achieving an accuracy rate of only 68.69% on the FaceForensics++ dataset, which contains various common types of face spoofing. Even with NeuralTextures, which has better visual effects, the accuracy rate for ordinary people is only about 50%, approaching random guessing. Given this, and considering that spoofed face videos are highly susceptible to defamation attacks and pose a significant threat to personal privacy, developing computer vision-based face spoofing video identification algorithms has great practical significance.

[0003] In fact, since 2018, deep learning-based face forgery detection technology has made significant progress, with existing methods achieving over 90% accuracy on publicly available face datasets such as FaceForensics++, Celeb-DF, and DeeperForensics. However, existing face forgery detection methods still suffer from insufficient robustness, mainly in the following two aspects:

[0004] 1) Insufficient robustness to video resolution: In practical applications, video data used to identify fake faces is often compressed to varying degrees by users or video platforms. Therefore, the model needs to perform well on videos of different resolutions. However, according to existing research, deep learning models trained on high-resolution data experience a significant performance drop when applied to low-resolution videos (accuracy decreases from approximately 95% to below 70%). Conversely, if the model is trained using low-resolution video data, it achieves less than ideal performance on both low-resolution and high-resolution videos (accuracy is approximately 85% in both cases). Simply mixing videos of different resolutions to train the model can address this issue to some extent, but the model's performance on both high-resolution and low-resolution videos is still lower than that achieved by training alone (using only high-resolution or low-resolution videos), resulting in unsatisfactory average performance.

[0005] 2) Insufficient robustness to different forgery methods: After several years of development, existing face forgery algorithms have evolved into many different approaches. The FaceForensics++ dataset alone contains various face forgery videos, including face replacement, face reconstruction, and face attribute manipulation. Because different forgery methods produce significantly different forgery flaws, the model cannot effectively distinguish between unfamiliar forgery methods, severely limiting the practical application of current methods. Summary of the Invention

[0006] In view of this, the purpose of this application is to propose a method and system for detecting fake face videos, which can specifically solve the existing problems.

[0007] To achieve the above objectives, this application proposes a method for detecting face spoofing videos, comprising:

[0008] 1) Preprocess the input video containing faces, and use a face recognition framework to detect and extract faces to obtain a frame-by-frame sequence of face image frames S;

[0009] 2) Using a convolutional neural network pre-trained on a large-scale image classification dataset, visual feature maps F of face images are generated from the extracted face frame sequences;

[0010] 3) Divide the feature map into 1×1 blocks, and then divide each feature map into blocks. Figure 1 The dimension is expanded into a sequence of token vectors T;

[0011] 4) Preprocess the token vector sequence by adding a classification token (T) to both the time and spatial dimensions. t and T s ), and add position encoding to obtain the token sequence I as input to the transformer;

[0012] 5) Use a video transformer neural network composed of multiple spatiotemporally separated transformer modules to process the token sequence;

[0013] 6) For the token sequence processed by the transformer network, a classification pooling layer is used to obtain the tokens for classification, and then a prediction head composed of fully connected layers is used to predict the true and false classification results.

[0014] In step 1), for the input video containing faces, a face recognition framework based on open source is used to detect and obtain bounding boxes for all faces in each frame. When more than one bounding box is detected, only the largest bounding box is used. Subsequently, face images are cropped based on the bounding boxes of each frame and uniformly scaled to 300×300 pixels. All face images constitute the frame sequence input to the subsequent model. The length of the frame sequence used is 6, so the shape of the frame sequence is 6×300×300×3.

[0015] In step 2), the convolutional neural network used is the Xception network pre-trained on ImageNet. For each input face image with a shape of 300×300×3, the network will output a feature map F with a shape of 19×19×2048.

[0016] In step 3), the feature map F is first divided into small blocks of shape 1×1×2048, and then each block is flattened into a 2048-dimensional vector as a token. For the entire frame sequence of length 6, all tokens will form a token sequence T of shape 6×361×2048.

[0017] In step 4), the token sequence T composed of all tokens will be categorized into spatial tokens T. t Token T categorized by time dimension s And a learnable positional encoding, to obtain the input token sequence I as the subsequent transformer network, which has a shape of 7×362×2048.

[0018] In step 5), the video transformer neural network used is based on spatiotemporal self-attention. Specifically, the token sequence I input to the transformer module is first mapped by a linear mapping layer (i.e., a fully connected layer) into Query, Key, and Value vectors Q, K, and V. These features are segmented into different heads (the number of heads is set to 8). Subsequently, self-attention calculations are performed on vectors Q, K, and V in the temporal and spatial dimensions. Specifically, for temporal self-attention, Q, K, and V corresponding to tokens in the same spatial location but different temporal locations will undergo self-attention operations, and these operations are performed synchronously in different spatial locations, as shown below:

[0019]

[0020] in Q represents the output of temporal self-attention at spatial location j. (：，j，：，：) K (：，j，：，：) and V(：，j，：，：) represents all tokens at spatial position j for Query, Key, and Value respectively, and D is the internal feature dimension.

[0021] Similarly, for spatial self-attention, all tokens at the same temporal position (i.e., within the same frame) but spatially different will have their corresponding Q, K, and V values ​​processed synchronously at different temporal positions, as represented by:

[0022]

[0023] Q represents the output of spatial self-attention at time position k (i.e., the k-th frame). (k，：，：，：) K (k，：，：，：) and V (k，：，：，：) These represent all tokens at time position k, representing the Query, Key, and Value respectively.

[0024] The video transformer neural network used introduces a residual subtraction mechanism. For an input token sequence I, residual subtraction means subtracting tokens in I that are spatially identical and temporally adjacent, to obtain the residual token sequence I', which is expressed as:

[0025] I′=cat((I (0：2，：，：) I (2：，：，：) -I (1：-1，：，：) ), dim=0)#

[0026] Where cat(A,B,dim=d) represents the concatenation operation of tensors A and B in the d-th dimension. (0∶2，：，：) For the 0th (category token) and 1st token at the time position that do not participate in the residual subtraction, I (2：，：，：) For the second to last token in the time position, I (1：-1，：，：) It refers to the first to second to last token in terms of time position.

[0027] The prediction head used in step 6) consists of three parts, with a pooling layer used to select class tokens (i.e., O) whose spatial and temporal positions are both 0. (0，0，：) The prediction results are output after layer normalization and an MLP head consisting of a fully connected network.

[0028] To achieve the above objectives, this application also proposes a face spoofing video detection system, comprising:

[0029] The frame sequence module is used to preprocess the input video containing faces, and uses a face recognition framework to detect and extract faces to obtain a frame-by-frame face image frame sequence S.

[0030] The visual feature map module is used to generate visual feature maps F of face images from extracted face frame sequences using a convolutional neural network pre-trained on a large-scale image classification dataset.

[0031] The slicing module is used to slice the feature map into 1×1 blocks and divide each block into... Figure 1 The dimension is expanded into a sequence of token vectors T;

[0032] The token sequence module is used to preprocess the token vector sequence, adding categorized tokens (T) in both the time and spatial dimensions. t and T s ), and add position encoding to obtain the token sequence I as input to the transformer;

[0033] The neural network module is used to process token sequences using a video transformer neural network composed of multiple spatiotemporally separated transformer modules;

[0034] The prediction and judgment module is used to obtain the tokens for classification from the token sequence processed by the transformer network using a classification pooling layer, and then use a prediction head composed of fully connected layers to predict the true or false classification result.

[0035] In summary, the advantages of this application and the user experience it brings are as follows:

[0036] First, because this invention uses the most advanced video transformer network structure, it can more effectively perform global modeling of temporal information in video, thus achieving higher accuracy and better robustness.

[0037] Second, the method proposed in this invention is based on a general video transformer structure, which is easier to understand, explain, and iteratively upgrade.

[0038] Third, this invention processes time and spatial attention independently, which is more suitable for the actual situation of face spoofing video detection, and therefore has a higher detection accuracy. Attached Figure Description

[0039] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0040] Figure 1 This is an overall flowchart of the method of the present invention.

[0041] Figure 2 This is a schematic diagram of the spatiotemporal separation self-attention used in the method of the present invention.

[0042] Figure 3 This is a schematic diagram of the residual subtraction mechanism used in the method of the present invention.

[0043] Figure 4 This is a schematic diagram illustrating the visualization results of the present invention as an example.

[0044] Figure 5 A schematic diagram of a face spoofing video detection system according to an embodiment of this application is shown.

[0045] Figure 6 A schematic diagram of the structure of an electronic device provided in one embodiment of this application is shown.

[0046] Figure 7 A schematic diagram of a storage medium provided in one embodiment of this application is shown. Detailed Implementation

[0047] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0048] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0049] The specific implementation steps of this invention are as follows:

[0050] Figure 1 This is a flowchart illustrating the face forgery detection method of the present invention. The specific implementation method is as follows:

[0051] 1) Preprocess the input video containing faces, and use a face recognition framework to detect and extract faces to obtain a frame-by-frame sequence of face image frames S;

[0052] 2) Using a convolutional neural network pre-trained on a large-scale image classification dataset, visual feature maps F of face images are generated from the extracted face frame sequences;

[0053] 3) Divide the feature map into 1×1 blocks, and then divide each feature map into blocks. Figure 1 The dimension is expanded into a sequence of token vectors T;

[0054] 4) Preprocess the token vector sequence by adding a classification token (T) to both the time and spatial dimensions. t and Ts ), and add position encoding to obtain the token sequence I as input to the transformer;

[0055] 5) Use a video transformer neural network composed of multiple spatiotemporally separated transformer modules to process the token sequence;

[0056] 6) For the token sequence processed by the transformer network, a classification pooling layer is used to obtain the tokens for classification, and then a prediction head composed of fully connected layers is used to predict the true and false classification results.

[0057] In step 1), for the input video containing faces, a face recognition framework based on open source is used to detect and obtain bounding boxes for all faces in each frame. When more than one bounding box is detected, only the largest bounding box is used. Subsequently, face images are cropped based on the bounding boxes of each frame and uniformly scaled to 300×300 pixels. All face images constitute the frame sequence input to the subsequent model. The length of the frame sequence used is 6, so the shape of the frame sequence is 6×300×300×3.

[0058] In step 2), the convolutional neural network used is the Xception network pre-trained on ImageNet. For each input face image with a shape of 300×300×3, the network will output a feature map F with a shape of 19×19×2048.

[0059] In step 3), the feature map F is first divided into small blocks of shape 1×1×2048, and then each block is flattened into a 2048-dimensional vector as a token. For the entire frame sequence of length 6, all tokens will form a token sequence T of shape 6×361×2048.

[0060] In step 4), the token sequence T composed of all tokens will be categorized into spatial tokens T. t Token T categorized by time dimension s And a learnable positional encoding, to obtain the input token sequence I as the subsequent transformer network, which has a shape of 7×362×2048.

[0061] The spatiotemporal separation self-attention mechanism of this invention is specifically described as follows: The basis for identifying forged face videos mainly includes spatial defects caused by image fusion and temporal defects caused by inter-frame discontinuities. These two types of defects originate from different principles and are therefore independent of each other. Based on this reason, this invention proposes a separate spatiotemporal self-attention mechanism based on the traditional video transformer to adapt to the task of identifying forged face videos, such as... Figure 2 As shown. The specific implementation is as follows:

[0062] The token sequence I input to the transformer module is first mapped by a linear mapping layer (i.e., a fully connected layer) into Query, Key, and Value vectors Q, K, and V. These features are then segmented into different heads, forming a tensor of shape (T+1)×(HW+1)×N×D. Here, T is the length of the input frame sequence, H is the height of the input frame, W is the width of the input frame, N is the number of heads for multi-head self-attention, and D is the internal feature dimension. Subsequently, self-attention can be calculated on vectors Q, K, and V.

[0063] In the case of temporal self-attention, tokens located in the same spatial position but different temporal positions will have their Q, K, and V values ​​processed simultaneously at different spatial positions, as shown below:

[0064]

[0065] in Q represents the output of temporal self-attention at spatial location j. (：，j，：，：) K (：，j，：，：) and V (k，：，：，：) These represent all tokens at position j in the query, key, and value space, respectively.

[0066] Similarly, for spatial self-attention, all tokens at the same temporal position (i.e., within the same frame) but spatially different will have their corresponding Q, K, and V values ​​processed synchronously at different temporal positions, as represented by:

[0067]

[0068] Similarly, Q represents the output of spatial self-attention at time position k (i.e., the k-th frame). (k，：，：，：) K (k，：，：，：) and V (k，：，：，：) These represent all tokens at time position k for Query, Key, and Value, respectively.

[0069] In the practical implementation of the temporal and spatial self-attention modules, the tensors Q, K, and V are simply rearranged into shapes N×(HW+1)×(T+1)×D and N×(T+1)×(HW+1)×D, respectively, and matrix multiplication is performed in the last two dimensions to achieve the above operation. The separate temporal and spatial self-attention mechanism proposed in this invention is not only more suitable for face forgery detection, but also reduces the computational complexity of self-attention matrix multiplication from O(T) to O(T). 2 H 2 W 2 ) significantly reduced to O(T 2 +H 2 W 2 This significantly reduces the amount of computation required for the model.

[0070] The residual subtraction mechanism of this invention is specifically described as follows: Since temporal information in video is extremely rich and redundant, to further encourage the temporal self-attention layer to focus on the discontinuities of faces between frames in the video frame sequence, this invention proposes a residual subtraction mechanism to further optimize the model. This mechanism is applied before the linear mapping layer of the temporal self-attention part, using the residuals of the inter-frame feature maps for mapping instead of the feature maps themselves when calculating vectors Q and V. Figure 3 As shown. Specifically, for an input token sequence I, residual subtraction is equivalent to subtracting tokens in I that have the same spatial location and adjacent temporal locations, resulting in the residual token sequence I', expressed as:

[0071] I′=cat((I (0：2，：，：) I (2：，：，：) -I (1-1，：，：) ), dim=0)#(4.3)

[0072] Where cat(A,B,dim=d) represents the concatenation operation of tensors A and B in the d-th dimension. (0：2，：，：） For the 0th (category token) and 1st token at the time position that do not participate in the residual subtraction, I (2：，：，：) For the second to last token in the time position, I (1：-1，：，：) These are the tokens from the first to the second-to-last in time. Subtracting them means subtracting all tokens in frame n from all tokens at the corresponding positions in frame (n-1). To avoid losing important spatial information, vector V is still obtained by mapping using the original token sequence I. This method proposed in this invention can effectively guide the temporal self-attention module to focus on the inter-frame discontinuities in the frame sequence, avoid being misled by redundant information, and further improve the robustness of the face forgery detection model against forgery methods.

[0073] The model interpretation method based on spatiotemporal dimension visualization of this invention is specifically described as follows: Model interpretability, that is, providing reasonable explanations for the conclusions drawn by the model, is of great significance for explainable AI and improving the performance and robustness of deep learning models. Furthermore, since existing fake face videos are visually extremely realistic, explaining the identification results can better help users understand why the video was judged as fake, which is beneficial for preventing fake face videos. Benefiting from the separate spatiotemporal self-attention proposed in this invention, the model prediction results can be explained separately in the temporal and spatial dimensions. Following the backpropagation formula based on deep Taylor expansion regarding the relationship between linear layers, residual connections, addition, layer normalization, and self-attention, the relationship between the temporal and spatial self-attention modules in the m-th transformer module can be calculated. Their shapes are N×(HW+1)×(T+1)×(T+1) and N×(T+1)×(HW+1)×(HW+1), respectively. Since face forgery video detection is a binary classification task, only class label 0, i.e., the class corresponding to the forgery label, needs backpropagation to calculate the relation and gradient. After calculating and saving the relation and gradient of each transformer module, the time and space visualization result U is calculated using the following formula. d (d∈{t,s}, representing time and space respectively):

[0074]

[0075]

[0076] Where I represents the identity matrix, E represents the Hadamard product operation, · represents the matrix multiplication operation. h This represents the mean along the head dimension. The subscripts (i, :, :) and (:, i, :, :) refer to... and U d It is calculated simultaneously for each spatial and temporal position i, and the superscript m represents the m-th position. This represents the relationship between the median value at the i-th temporal or spatial location and the model's prediction result within the temporal or spatial attention layer of the m-th transformer module. This represents the gradient of the spatiotemporal self-attention layer in the m-th transformer module, i.e.:

[0077]

[0078] in and This represents the values ​​of Query and Key in the spatiotemporal attention layer of the m-th transformer module. The visualization result U obtained from the above steps in the temporal and spatial dimensions is... t and U s These are tensors of shapes (HW+1)×(T+1)×(T+1) and (T+1)×(HW+1)×(HW+1), respectively. Where U t It can be viewed as a series of matrices at every spatial location, and similarly, U s This can be viewed as a series of matrices at each time point. Each row in the matrix represents the relevance of a token to all other tokens. Therefore, the first row of the matrix, representing the relevance of the classification token to all other tokens, can be used for visualization. This is because the classification token extracts information from all tokens through self-attention for classification; thus, its relevance to other tokens can be considered the magnitude of these tokens' influence on the result. Therefore, when selecting U... d The first row of all matrices, after removing matrices consisting only of relationships between classification tokens (these relationships are meaningless in explaining the impact of image content on the discrimination results), U t and U s The shapes are HW×1×T and T×1×HW, respectively. Finally, U t and U s Rearrange the image to T×H×W, then upsample it to the original image size (300×300) using bilinear interpolation to obtain the final visualization result. An example visualization is shown below. Figure 4 As shown.

[0079] Tables 1 and 2 show the performance comparison between the algorithm of this invention and other algorithms.

[0080] Table 1 compares the performance of the algorithm of this invention with other algorithms on the publicly available face forgery detection datasets FaceForensics++, Celeb-DF (CDF), and DFDC.

[0081]

[0082] Table 2. Performance comparison of the algorithm of this invention with other algorithms in cross-dataset scenarios.

[0083]

[0084]

[0085] The application provides a face spoofing video detection system, which is used to execute the face spoofing video detection method described in the above embodiments, such as... Figure 5 As shown, the system includes:

[0086] The frame sequence module 501 is used to preprocess the input video containing human faces, and use a face recognition framework to detect and extract human faces to obtain a frame-by-frame human face image frame sequence S.

[0087] Visual feature map module 502 is used to generate visual feature maps F of face images from extracted face frame sequences using a convolutional neural network pre-trained on a large-scale image classification dataset.

[0088] The slicing module 503 is used to slice the feature map into 1×1 blocks and divide each feature map into blocks. Figure 1 The dimension is expanded into a sequence of token vectors T;

[0089] The token sequence module 504 is used to preprocess the token vector sequence, adding categorized tokens (T) in both the time and spatial dimensions. t and T s ), and add position encoding to obtain the token sequence I as input to the transformer;

[0090] Neural network module 505 is used to process token sequences using a video transformer neural network composed of multiple spatiotemporally separated transformer modules;

[0091] The prediction and judgment module 506 is used to obtain the tokens for classification from the token sequence processed by the transformer network using a classification pooling layer, and then use a prediction head composed of fully connected layers to predict the true or false classification result.

[0092] The face spoofing video detection system and the face spoofing video detection method provided in the above embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0093] This application also provides an electronic device corresponding to the face spoofing video detection method provided in the foregoing embodiments, for executing the face spoofing video detection method. This application does not limit the scope of the embodiments.

[0094] Please refer to Figure 6 This illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 6As shown, the electronic device 20 includes: a processor 200, a memory 201, a bus 202, and a communication interface 203. The processor 200, the communication interface 203, and the memory 201 are connected via the bus 202. The memory 201 stores a computer program that can run on the processor 200. When the processor 200 runs the computer program, it executes the face spoofing video detection method provided in any of the foregoing embodiments of this application.

[0095] The memory 201 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 203 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0096] Bus 202 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store programs. After receiving an execution instruction, the processor 200 executes the program. The face spoofing video detection method disclosed in any of the foregoing embodiments of this application can be applied to the processor 200, or implemented by the processor 200.

[0097] The processor 200 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 200 or by instructions in software form. The processor 200 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 201. The processor 200 reads the information in memory 201 and, in conjunction with its hardware, completes the steps of the above method.

[0098] The electronic device provided in this application embodiment and the face spoofing video detection method provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0099] This application also provides a computer-readable storage medium corresponding to the face spoofing video detection method provided in the foregoing embodiments. Please refer to... Figure 7 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the face spoofing video detection method provided in any of the foregoing embodiments.

[0100] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0101] The computer-readable storage medium provided in the above embodiments of this application and the face spoofing video detection method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0102] It should be noted that:

[0103] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, this application is not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application.

[0104] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0105] Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this method of disclosure should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0106] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0107] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0108] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the virtual machine creation system according to the embodiments of this application. This application can also be implemented as a device or system program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0109] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several systems, several of these systems may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0110] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for detecting fake faces in videos, based on a spatiotemporal separation self-attention video transformer, characterized in that, Includes the following steps: 1) Preprocess the input video containing faces, and use a face recognition framework to detect and extract faces to obtain a frame-by-frame sequence of face image frames; 2) Using a convolutional neural network pre-trained on a large-scale image classification dataset, generate visual feature maps of face images from the face image frame sequence; 3) Divide the visual feature map into 1×1 blocks, and unfold each block of the visual feature map into a one-dimensional token vector sequence; 4) Preprocess the token vector sequence by adding classification tokens in the time and spatial dimensions respectively, and adding position encoding to obtain the token sequence as input to the transformer; 5) The token sequence is processed using a video transformer neural network composed of multiple spatiotemporally separated transformer modules; in step 5), the video transformer neural network used is based on spatiotemporally separated self-attention; specifically, the token sequence I input to the transformer module is first mapped by a linear mapping layer into Query, Key, and Value vectors Q, K, V. These features are segmented into different heads, and then self-attention calculations are performed on vectors Q, K, and V in the temporal and spatial dimensions; wherein, for temporal self-attention, self-attention operations are performed on Q, K, and V corresponding to tokens that are in the same spatial position but different temporal positions, and these operations are performed synchronously in different spatial positions, as shown below: ， in This represents the output of the temporal self-attention at spatial location j. , and These represent all tokens at spatial position j for Query, Key, and Value, respectively, and D is the internal feature dimension. For spatial self-attention, the Q, K, and V values ​​of all tokens that are at the same temporal position but spatially different are processed for self-attention, and this is done synchronously at different temporal positions, as represented by: ， This represents the output of spatial self-attention at time position k, i.e., the k-th frame. , and These represent all tokens at time position k, representing the Query, Key, and Value respectively. The video transformer neural network used introduces a residual subtraction mechanism. For an input token sequence I, residual subtraction means subtracting tokens in I that are spatially identical and temporally adjacent, to obtain the residual token sequence I', which is expressed as: ， The function `cat(A,B,dim=d)` concatenates tensors A and B along the d-th dimension. For the 0th and 1st category tokens at the time position that do not participate in the residual subtraction, For the second to last token in terms of time position, The tokens are the first to the second to last token in terms of time position; 6) For the token sequence processed by the video transformer neural network, a classification pooling layer is used to obtain the tokens for classification, and then a prediction head composed of fully connected layers is used to predict the true and false classification results.

2. The method according to claim 1, characterized in that, Step 1) specifically includes the following steps: For input videos containing faces, an open-source face recognition framework is used to detect and obtain bounding boxes of all faces in each frame. When more than one bounding box of a face is detected, only the largest bounding box is used. The face image is cropped from the bounding box of each frame and uniformly scaled to a size of 300×300. All the face images constitute the frame sequence input to the subsequent model.

3. The method according to claim 1, characterized in that, In step 2), the convolutional neural network used is the Xception network pre-trained on ImageNet. For each input face image with a shape of 300×300×3, the network will output a feature map with a shape of 19×19×2048.

4. The method according to claim 1, characterized in that, In step 3), the feature map is first divided into small blocks of shape 1×1×2048, and then each small block is flattened into a 2048-dimensional vector as a token; for the entire frame sequence of length 6, all tokens constitute a token sequence of shape 6×361×2048.

5. The method according to claim 1, characterized in that, In step 4), the token sequence consisting of all tokens is categorized into spatial tokens T. t And the time-based classification token T s And a learnable positional encoding, resulting in a sequence of tokens with a shape of 7×362×2048 that serves as the input to the subsequent transformer network.

6. The method according to claim 1, characterized in that, The prediction head used in step 6) consists of three parts: a pooling layer to select classification tokens whose spatial and temporal positions are both 0, a layer normalization layer, and an MLP head composed of a fully connected network to output the prediction result.

7. A face forgery video detection system, using the method described in any one of claims 1-6, characterized in that, include: The frame sequence module is used to preprocess the input video containing faces, and uses a face recognition framework to detect and extract faces to obtain a frame-by-frame sequence of face images. A visual feature map module is used to generate visual feature maps of face images from the face image frame sequence using a convolutional neural network pre-trained on a large-scale image classification dataset. The slicing module is used to slice the visual feature map into 1×1 blocks and unfold each visual feature map block into a one-dimensional sequence of token vectors. The token sequence module is used to preprocess the token vector sequence by adding categorized tokens in the time and spatial dimensions and adding position encoding to obtain the token sequence as input to the transformer. A neural network module is used to process the token sequence using a video transformer neural network composed of multiple spatiotemporally separated transformer modules; The prediction and judgment module is used to obtain the tokens for classification from the token sequence processed by the video transformer neural network using a classification pooling layer, and then use a prediction head composed of fully connected layers to predict the true and false classification results.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the method as described in any one of claims 1-6.

Images