False face video detection method, device, equipment and medium

Abstract

The invention discloses a forged face video detection method and device, equipment and a medium, and relates to the technical field of video detection, and the method comprises the steps: carrying out the sampling of a to-be-detected video, and obtaining a plurality of video segments; wherein each video clip comprises a plurality of continuous video frames; inputting the plurality of video clips into a first detection model for detection, and outputting a forgery probability; wherein the first detection model is provided with a space Transform module and a time Transform module in sequence; each of the space Transform module and the time Transform module comprises a plurality of same coding layers; the coding layer is sequentially provided with an attention module and a feedforward network module; the attention module converts a query sequence and a value sequence calculated through input into attention output based on a symmetric global weighted attention mechanism; and obtaining a detection result according to the forgery probability. According to the invention, the detection efficiency of the fake face video can be improved.

Description

Technical Field

[0001] This application relates to the field of video detection technology, and in particular to a method, apparatus, device and medium for detecting fake face videos. Background Technology

[0002] In recent years, with the continuous advancement of deepfake technology—a technique based on deep learning models to generate realistic artificial content—the generated fake face videos have become visually indistinguishable from real ones, posing a potential security threat to social information security. Therefore, it is necessary to research and develop efficient and accurate fake face video detection technologies to counter deepfake technology and safeguard information security.

[0003] Current fake face video detection technologies, such as using dynamic differences for fake face detection, freezing spatiotemporal weights during training for fake face detection, or combining illumination information and spatiotemporal inconsistency for fake face detection, all improve detection accuracy by employing different technical means. However, these detection methods generally have the following two drawbacks: (1) High computational complexity: Existing Transformer-based models use a standard self-attention mechanism, which requires repetitive and complex calculations on the query sequence, key sequence, and value sequence generated from the input data. Its computational complexity is proportional to the square of the length of the input sequence. (1) Video data contains a large number of frames and pixels, and the input sequence is extremely long, resulting in huge computational overhead for the model and difficulty in achieving the computational efficiency required for real-time operation; (2) The parameter scale is huge: in order to pursue higher accuracy, the model design is becoming more and more complex, especially in the feedforward network (FFN) part, which adds a large number of parameters, resulting in a bloated model and high memory consumption, thereby slowing down the model's computational efficiency. These two shortcomings seriously hinder the actual deployment and application of fake face detection technology on resource-constrained terminals such as mobile phones and embedded devices. Therefore, how to improve the efficiency of fake face video detection while maintaining the accuracy of fake face video detection is still a technical problem that needs to be solved in the existing technology. Summary of the Invention

[0004] This application provides a method, apparatus, device, and medium for detecting fake face videos, in order to solve the technical problem that the detection efficiency of existing fake face detection methods is not up to expectations.

[0005] According to a first aspect of the embodiments of this application, a method for detecting fake face videos is provided, comprising: The video to be tested is sampled to obtain multiple video segments; each video segment contains multiple consecutive video frames. The multiple video segments are input into a preset first detection model for detection, and the forgery probability of the video to be detected is output. The first detection model sequentially includes a spatial Transformer module and a temporal Transformer module. Both the spatial and temporal Transformer modules contain multiple identical encoding layers. Each encoding layer sequentially includes an attention module and a feedforward network module. The attention module, based on a symmetric global weighted attention mechanism, transforms the query sequence and value sequence calculated from the input data into an attention output. The detection result of the video to be detected is obtained based on the forgery probability.

[0006] This application first samples the video to be detected to obtain multiple video segments, and each video segment contains multiple consecutive video frames, ensuring comprehensive sampling and providing data support for maintaining detection accuracy. Then, the multiple video segments are input into a first detection model for detection. The first detection model is configured with spatial Transformer modules and temporal Transformer modules, each containing multiple identical coding layers. Each coding layer is configured with an attention module and a feedforward network module. The attention module is based on a symmetric global weighted attention mechanism, transforming the query sequence and value sequence calculated from the input data into attention outputs. The first detection model outputs the forgery probability of the video to be detected, and the detection result of the video to be detected is obtained based on the forgery probability. By setting spatial and temporal Transformers, the spatial and temporal features of the video to be detected can be extracted respectively, and the combination of the two can ensure detection accuracy. By specially configuring the attention module of the coding layer, based on the symmetric global weighted attention mechanism, the query sequence and value sequence are directly transformed without calculating the key sequence in the existing standard attention mechanism, which can reduce the computational dimension and the number of computational parameters, thereby reducing computational complexity and improving computational efficiency, thus improving the detection efficiency of forged faces.

[0007] In some embodiments of this application, sampling the video to be detected to obtain multiple video segments specifically includes: The video to be detected is randomly sampled to obtain multiple original video segments; wherein each original video segment consists of multiple consecutive original video frames; The face region of each original video frame in each original video segment is detected, and the face region is cropped and subjected to affine transformation to align the face key points of each original video frame in each original video segment, resulting in multiple video segments.

[0008] This application first randomly samples the video to be detected to obtain multiple original video segments, each containing multiple consecutive original video frames. Then, it detects, crops, and performs affine transformations on the face regions of each original video frame to obtain corresponding multiple video segments. By randomly sampling the video, the randomness of the input can be guaranteed, thus ensuring that the sampled part fully reflects the overall content of the video, thereby providing corresponding data support for maintaining the accuracy of subsequent detection. By aligning the facial key points of each original video frame in each original segment, the consistency of the input can be guaranteed, avoiding recognition errors caused by misaligned faces, thereby ensuring the accuracy of subsequent detection.

[0009] In some embodiments of this application, the first detection model further includes a feature extraction module before the spatial Transformer module and a classifier after the temporal Transformer module; the step of inputting the plurality of video segments into the preset first detection model for detection and outputting the forgery probability of the video to be detected specifically includes: The feature extraction module extracts features from each video frame of each video segment. During extraction, a multi-dimensional feature map of the video frame is obtained based on depthwise separable convolution, and video frames belonging to the same video segment are aggregated to obtain the intra-frame feature sequence of each video segment. The spatial Transformer module iteratively processes the intra-frame feature sequence of each video segment based on the multiple coding layers. During iteration, the query sequence and value sequence calculated from the intra-frame feature sequence are transformed into attention output according to the attention module of the current coding layer. The attention output is then calibrated according to the feedforward network module of the current coding layer to obtain the intra-frame feature sequence for the next iteration. This process continues until each coding layer of the spatial Transformer module has been processed, resulting in the spatial feature sequence of each video segment. The spatial feature sequences of each video segment are then aggregated to obtain the inter-frame feature sequence of the video to be detected. The temporal Transformer module iteratively processes the inter-frame feature sequence based on the multiple coding layers. During iteration, the query sequence and value sequence calculated from the inter-frame feature sequence are transformed into attention outputs according to the attention module of the current coding layer. The attention outputs are then calibrated according to the feedforward network module of the current coding layer to obtain the inter-frame feature sequence for the next iteration. This process continues until each coding layer of the temporal Transformer module has been processed, resulting in the spatiotemporal feature sequence of the video to be detected. The classifier aggregates the spatiotemporal feature sequences into a decision vector, and performs label prediction based on the decision vector to obtain and output the forgery probability of the video to be detected.

[0010] This application first extracts intra-frame feature sequences for each video segment using a feature extraction module based on depthwise separable convolution, which can fully extract features through depthwise separable convolution. Then, a spatial Transformer module iterates over the intra-frame feature sequences of each video segment. During iteration, an attention module transforms the query and value sequences corresponding to the intra-frame feature sequences into attention outputs, and inter-frame feature sequences are obtained by aggregating the output data. This allows for direct transformation of the query and value sequences without calculating the key sequence in existing standard attention mechanisms, reducing computational dimensionality and the number of computational parameters, thereby reducing computational complexity. Next, a temporal Transformer module performs the same transformation on the inter-frame feature sequences of the video to be detected as the spatial Transformer module, obtaining the spatiotemporal feature sequences of the video to be detected. Spatial and temporal features can be extracted separately through spatial and temporal Transformers, and then combined for detection to ensure detection accuracy. Finally, a classifier aggregates the spatiotemporal feature sequences into decision vectors and performs label prediction to obtain the corresponding forgery probability. This allows for further processing of the extracted spatiotemporal features to obtain the desired output type, which meets practical needs.

[0011] In some embodiments of this application, the step of extracting features from each video frame of each video segment, wherein the extraction is performed by obtaining a multidimensional feature map of the video frame based on depthwise separable convolution, and video frames belonging to the same video segment are aggregated to obtain an intra-frame feature sequence for each video segment, specifically includes: Based on the multiple depthwise separable convolutional modules of the feature extraction module, feature extraction is performed on each video frame of each video segment to obtain a multidimensional feature map of each video frame of each video segment; By stretching the dimensions, the multidimensional feature map of each video frame of each video segment is converted into a corresponding one-dimensional vector. The one-dimensional vectors of video frames belonging to the same video segment are then aggregated and arranged according to the timestamp order of the video frames during aggregation, resulting in the intra-frame feature sequence of each video segment.

[0012] This application first performs feature extraction based on multiple depthwise separable convolutional modules to obtain multidimensional feature maps for each video frame of each video segment. This allows for multidimensional feature extraction through multiple depthwise separable convolutional modules, fully recognizing and acquiring the features of the input frame. Then, the multidimensional feature maps are converted into corresponding one-dimensional vectors through dimensional stretching and aggregated. During aggregation, the vectors are arranged according to the timestamp order of the video frames to obtain the intra-frame feature sequence of each video segment. This approach can maintain the multidimensional features recognized by depthwise separable convolution while preserving the format of the intra-frame feature sequence through the timestamp order, providing an adapted input format for subsequent temporal feature extraction.

[0013] In some embodiments of this application, the transformation of the query sequence and value sequence calculated from the intra-frame feature sequence by the attention module of the current coding layer into attention output specifically includes: Linear projection is performed on the intra-frame feature sequence to obtain the corresponding query sequence and value sequence; Global projection is performed on each query token of the query sequence and each value token of the value sequence to obtain the importance score of each query token and the importance score of each value token. Based on the importance score of each query token, the query sequence is weighted and summed to obtain the query global context vector corresponding to the query sequence. Based on the importance score of each value token, the value sequence is weighted and summed to obtain the value global context vector corresponding to the value sequence. The query global context vector is fused with the value sequence to obtain a first fusion matrix, the value global context vector is fused with the query sequence to obtain a second fusion matrix, and the first fusion matrix and the second fusion matrix are integrated to obtain the attention output.

[0014] This application first performs linear projection on the intra-frame feature sequence to obtain the query sequence and value sequence. Then, it performs global projection on the corresponding query token and value token to obtain the importance score of each query token and value token. Subsequently, it performs weighted summation on the query sequence and value sequence to obtain the corresponding global context vector. By performing corresponding projection transformation on the query sequence and value sequence through the global context vector, it can replace the role of the key sequence in the existing standard attention mechanism, reduce the computational dimension and the number of computational parameters, thereby reducing computational complexity and improving computational efficiency. Finally, it performs symmetric cross-fusion on the query sequence, value sequence and the corresponding global context vector, which can fully fuse to generate the corresponding attention and maintain the corresponding detection accuracy.

[0015] In some embodiments of this application, the step of calibrating the attention output based on the feedforward network module of the current coding layer to obtain the intra-frame feature sequence for the next iteration specifically includes: The attention output is subjected to a higher-dimensional linear mapping and activated by an activation function with leakage to obtain a first intermediate feature. The first intermediate feature is then subjected to global average pooling to obtain a second intermediate feature. The second intermediate feature is sequentially subjected to a dimensionality reduction linear mapping and a dimensionality increase linear mapping to extract channel weights, resulting in a channel weight vector; wherein the dimension of the channel weight vector is the same as that of the second intermediate feature. The channel weight vector is fused with the first intermediate feature to obtain the intra-frame feature sequence for the next iteration.

[0016] This application first performs a dimensionality-upgrading linear mapping on the attention output and activates it using an activation function with leakage. Then, global average pooling is performed to obtain the second intermediate feature. Next, dimensionality-downgrading and dimensionality-upgrading linear mappings are performed sequentially to extract channel weights. Finally, the channel weights and the first intermediate feature are fused to obtain the intra-frame feature sequence. By extracting channel weights and fusing them with the corresponding features, channels with high channel weights in the features can be enhanced, while channels with low channel weights can be suppressed, thereby improving feature saliency and maintaining the corresponding detection accuracy.

[0017] In some embodiments of this application, the step of aggregating the spatiotemporal feature sequence into a decision vector, and performing label prediction based on the decision vector to obtain and output the forgery probability of the video to be detected specifically includes: Global pooling is performed on the spatiotemporal feature sequence to aggregate the feature information in the spatiotemporal feature sequence and obtain a decision vector; Based on the fully connected layer of the classifier, label prediction is performed on the decision vector to obtain and output the forgery probability of the video to be detected.

[0018] This application first performs global pooling on the spatiotemporal feature sequence to obtain the decision vector, and then predicts the corresponding forgery probability based on the fully connected layer. It can aggregate the feature information of the spatiotemporal feature sequence and further process it into the expected output type, which meets the actual needs.

[0019] According to a second aspect of the embodiments of this application, a fake face video detection device is provided, including a video sampling module, a model detection module, and a result output module; The video sampling module is used to sample the video to be detected to obtain multiple video segments; wherein each video segment contains multiple consecutive video frames; The model detection module is used to input the multiple video segments into a preset first detection model for detection and output the forgery probability of the video to be detected. The first detection model sequentially includes a spatial Transformer module and a temporal Transformer module. Both the spatial and temporal Transformer modules contain multiple identical encoding layers. Each encoding layer sequentially includes an attention module and a feedforward network module. The attention module, based on a symmetric global weighted attention mechanism, transforms the query sequence and value sequence calculated from the input data into an attention output. The result output module is used to obtain the detection result of the video to be detected based on the forgery probability.

[0020] In some embodiments of this application, the video sampling module includes a random sampling unit and a segment processing unit; The random sampling unit is used to randomly sample the video to be detected to obtain multiple original video segments; wherein each original video segment consists of multiple consecutive original video frames. The segment processing unit is used to detect the face region of each original video frame in each original video segment, crop the face region, and perform an affine transformation to align the face key points of each original video frame in each original video segment, thereby obtaining multiple video segments.

[0021] In some embodiments of this application, the first detection model is further provided with a feature extraction module before the spatial Transformer module and a classifier after the temporal Transformer module; the model detection module includes a feature extraction unit, a spatial processing unit, a temporal processing unit and a classification prediction unit; The feature extraction unit is used to extract features from each video frame of each video segment through the feature extraction module. During extraction, a multi-dimensional feature map of the video frame is obtained based on depthwise separable convolution, and video frames belonging to the same video segment are aggregated to obtain the intra-frame feature sequence of each video segment. The spatial processing unit is used to iteratively process the intra-frame feature sequence of each video segment based on the multiple coding layers through the spatial Transformer module. During iteration, the query sequence and value sequence calculated from the intra-frame feature sequence are transformed into attention output according to the attention module of the current coding layer, and the attention output is calibrated according to the feedforward network module of the current coding layer to obtain the intra-frame feature sequence for the next iteration, until each coding layer of the spatial Transformer module has been processed to obtain the spatial feature sequence of each video segment, and the spatial feature sequences of each video segment are aggregated to obtain the inter-frame feature sequence of the video to be detected. The temporal processing unit is used to iteratively process the inter-frame feature sequence based on the multiple coding layers through the temporal Transformer module. During iteration, the query sequence and value sequence calculated by the inter-frame feature sequence are transformed into attention output according to the attention module of the current coding layer, and the attention output is calibrated according to the feedforward network module of the current coding layer to obtain the inter-frame feature sequence for the next iteration, until each coding layer of the temporal Transformer module has been processed to obtain the spatiotemporal feature sequence of the video to be detected. The classification and prediction unit is used to aggregate the spatiotemporal feature sequence into a decision vector through the classifier, and perform label prediction based on the decision vector to obtain and output the forgery probability of the video to be detected.

[0022] In some embodiments of this application, the step of extracting features from each video frame of each video segment, wherein the extraction is performed by obtaining a multidimensional feature map of the video frame based on depthwise separable convolution, and video frames belonging to the same video segment are aggregated to obtain an intra-frame feature sequence for each video segment, specifically includes: Based on the multiple depthwise separable convolutional modules of the feature extraction module, feature extraction is performed on each video frame of each video segment to obtain a multidimensional feature map of each video frame of each video segment; By stretching the dimensions, the multidimensional feature map of each video frame of each video segment is converted into a corresponding one-dimensional vector. The one-dimensional vectors of video frames belonging to the same video segment are then aggregated. During aggregation, the vectors are arranged according to the timestamp order of the video frames to obtain the intra-frame feature sequence of each video segment.

[0023] In some embodiments of this application, the transformation of the query sequence and value sequence calculated from the intra-frame feature sequence by the attention module of the current coding layer into attention output specifically includes: Linear projection is performed on the intra-frame feature sequence to obtain the corresponding query sequence and value sequence; Global projection is performed on each query token of the query sequence and each value token of the value sequence to obtain the importance score of each query token and the importance score of each value token. Based on the importance score of each query token, the query sequence is weighted and summed to obtain the query global context vector corresponding to the query sequence. Based on the importance score of each value token, the value sequence is weighted and summed to obtain the value global context vector corresponding to the value sequence. The query global context vector is fused with the value sequence to obtain a first fusion matrix, the value global context vector is fused with the query sequence to obtain a second fusion matrix, and the first fusion matrix and the second fusion matrix are integrated to obtain the attention output.

[0024] In some embodiments of this application, the step of calibrating the attention output based on the feedforward network module of the current coding layer to obtain the intra-frame feature sequence for the next iteration specifically includes: The attention output is subjected to a higher-dimensional linear mapping and activated by an activation function with leakage to obtain a first intermediate feature. The first intermediate feature is then subjected to global average pooling to obtain a second intermediate feature. The second intermediate feature is sequentially subjected to a dimensionality reduction linear mapping and a dimensionality increase linear mapping to extract channel weights, resulting in a channel weight vector; wherein the dimension of the channel weight vector is the same as that of the second intermediate feature. The channel weight vector is fused with the first intermediate feature to obtain the intra-frame feature sequence for the next iteration.

[0025] In some embodiments of this application, the step of aggregating the spatiotemporal feature sequence into a decision vector, and performing label prediction based on the decision vector to obtain and output the forgery probability of the video to be detected specifically includes: Global pooling is performed on the spatiotemporal feature sequence to aggregate the feature information in the spatiotemporal feature sequence and obtain a decision vector; Based on the fully connected layer of the classifier, label prediction is performed on the decision vector to obtain and output the forgery probability of the video to be detected.

[0026] This application first samples the video to be detected to obtain multiple video segments, and each video segment contains multiple consecutive video frames, ensuring comprehensive sampling and providing data support for maintaining detection accuracy. Then, the multiple video segments are input into a first detection model for detection. The first detection model is configured with spatial Transformer modules and temporal Transformer modules, each containing multiple identical coding layers. Each coding layer is configured with an attention module and a feedforward network module. The attention module is based on a symmetric global weighted attention mechanism, transforming the query sequence and value sequence calculated from the input data into attention outputs. The first detection model outputs the forgery probability of the video to be detected, and the detection result of the video to be detected is obtained based on the forgery probability. By setting spatial and temporal Transformers, the spatial and temporal features of the video to be detected can be extracted respectively, and the combination of the two can ensure detection accuracy. By specially configuring the attention module of the coding layer, based on the symmetric global weighted attention mechanism, the query sequence and value sequence are directly transformed without calculating the key sequence in the existing standard attention mechanism, which can reduce the computational dimension and the number of computational parameters, thereby reducing computational complexity and improving computational efficiency, thus improving the detection efficiency of forged faces.

[0027] According to a third aspect of the embodiments of this application, a computer device is provided, comprising: a processor; a memory; and a computer program stored in the memory and configured to be executed by the processor; wherein the processor executes the computer program to implement a method for detecting fake face videos according to this application.

[0028] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute a method for detecting fake face videos according to this application. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating a method for detecting fake face videos according to certain embodiments of this application. Figure 2 This is a schematic diagram illustrating the structure of an attention module based on a symmetric global weighted attention mechanism, as shown in certain embodiments of this application. Figure 3 This is a schematic diagram of the structure of a feedforward neural network shown in some embodiments of this application; Figure 4 This is a module structure diagram of a fake face video detection device shown in some embodiments of this application. Detailed Implementation

[0030] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below in conjunction with the accompanying drawings are exemplary and are only used to explain some embodiments of this application, and should not be construed as limiting the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments shown in this application without inventive effort are within the protection scope of this application.

[0031] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, unless otherwise explicitly specified, "a plurality of" or "several" means two or more.

[0032] Existing fake face video detection technologies, such as using dynamic differences for fake face detection, freezing spatiotemporal weights during training for fake face detection, or combining illumination information and spatiotemporal inconsistency for fake face detection, have the following drawbacks: (1) High computational complexity: Existing Transformer-based models use a standard self-attention mechanism, which requires repetitive and complex calculations on the query sequence, key sequence, and value sequence generated from the input data. Its computational complexity is proportional to the square of the length of the input sequence. (1) Video data contains a large number of frames and pixels, and the input sequence is extremely long, resulting in huge computational overhead for the model and difficulty in achieving the computational efficiency required for real-time operation; (2) The parameter scale is huge: in order to pursue higher accuracy, the model design is becoming more and more complex, especially in the feedforward network (FFN) part, which adds a large number of parameters, resulting in a bloated model and high memory consumption, thereby slowing down the model's computational efficiency. These two shortcomings seriously hinder the actual deployment and application of fake face detection technology on resource-constrained terminals such as mobile phones and embedded devices. Therefore, how to improve the efficiency of fake face video detection while maintaining the accuracy of fake face video detection is still a technical problem that needs to be solved in the existing technology.

[0033] Based on the above technical background, please refer to Figure 1 This application provides a method for detecting fake face videos, including steps S101 to S103, each step of which is as follows: Step S101: Sample the video to be detected to obtain multiple video segments; wherein each video segment contains multiple consecutive video frames.

[0034] In some embodiments of this application, sampling the video to be detected to obtain multiple video segments specifically includes: The video to be detected is randomly sampled to obtain multiple original video segments; wherein each original video segment consists of multiple consecutive original video frames; The face region of each original video frame in each original video segment is detected, and the face region is cropped and subjected to affine transformation to align the face key points of each original video frame in each original video segment, resulting in multiple video segments.

[0035] Specifically, the preferred number of original video segments obtained by random sampling is 4, and the preferred number of consecutive original video frames for each original video segment is 8; the preferred method for detecting and cropping the face region is to perform face detection and cropping using MTCNN (Multi-task convolutional neural network), and the preferred cropping resolution is 300×300; the facial key points of the face region include, but are not limited to, the eyes, nose, and mouth, as well as the feature points used to indicate them.

[0036] It is easy to understand that any video frame mentioned or appearing in this application refers to a single image displayed in the video, that is, each video frame is an independent image, and more specifically, an independent image of a human face.

[0037] This application first randomly samples the video to be detected to obtain multiple original video segments, each containing multiple consecutive original video frames. Then, it detects, crops, and performs affine transformations on the face regions of each original video frame to obtain corresponding multiple video segments. By randomly sampling the video, the randomness of the input can be guaranteed, thus ensuring that the sampled part fully reflects the overall content of the video, thereby providing corresponding data support for maintaining the accuracy of subsequent detection. By aligning the facial key points of each original video frame in each original segment, the consistency of the input can be guaranteed, avoiding recognition errors caused by misaligned faces, thereby ensuring the accuracy of subsequent detection.

[0038] Step S102: Input the multiple video segments into a preset first detection model for detection, and output the forgery probability of the video to be detected; wherein, the first detection model is configured with a spatial Transformer module and a temporal Transformer module in sequence; both the spatial Transformer module and the temporal Transformer module contain multiple identical encoding layers; the encoding layers are configured with an attention module and a feedforward network module in sequence; the attention module is based on a symmetric global weighted attention mechanism, which transforms the query sequence and value sequence calculated from the input data into attention output.

[0039] In some embodiments of this application, the first detection model further includes a feature extraction module before the spatial Transformer module and a classifier after the temporal Transformer module; the step of inputting the plurality of video segments into the preset first detection model for detection and outputting the forgery probability of the video to be detected specifically includes: The feature extraction module extracts features from each video frame of each video segment. During extraction, a multi-dimensional feature map of the video frame is obtained based on depthwise separable convolution, and video frames belonging to the same video segment are aggregated to obtain the intra-frame feature sequence of each video segment. The spatial Transformer module iteratively processes the intra-frame feature sequence of each video segment based on the multiple coding layers. During iteration, the query sequence and value sequence calculated from the intra-frame feature sequence are transformed into attention output according to the attention module of the current coding layer. The attention output is then calibrated according to the feedforward network module of the current coding layer to obtain the intra-frame feature sequence for the next iteration. This process continues until each coding layer of the spatial Transformer module has been processed, resulting in the spatial feature sequence of each video segment. The spatial feature sequences of each video segment are then aggregated to obtain the inter-frame feature sequence of the video to be detected. The temporal Transformer module iteratively processes the inter-frame feature sequence based on the multiple coding layers. During iteration, the query sequence and value sequence calculated from the inter-frame feature sequence are transformed into attention outputs according to the attention module of the current coding layer. The attention outputs are then calibrated according to the feedforward network module of the current coding layer to obtain the inter-frame feature sequence for the next iteration. This process continues until each coding layer of the temporal Transformer module has been processed, resulting in the spatiotemporal feature sequence of the video to be detected. The classifier aggregates the spatiotemporal feature sequences into a decision vector, and performs label prediction based on the decision vector to obtain and output the forgery probability of the video to be detected.

[0040] This application first extracts intra-frame feature sequences for each video segment using a feature extraction module based on depthwise separable convolution, which can fully extract features through depthwise separable convolution. Then, a spatial Transformer module iterates over the intra-frame feature sequences of each video segment. During iteration, an attention module transforms the query and value sequences corresponding to the intra-frame feature sequences into attention outputs, and inter-frame feature sequences are obtained by aggregating the output data. This allows for direct transformation of the query and value sequences without calculating the key sequence in existing standard attention mechanisms, reducing computational dimensionality and the number of computational parameters, thereby reducing computational complexity. Next, a temporal Transformer module performs the same transformation on the inter-frame feature sequences of the video to be detected as the spatial Transformer module, obtaining the spatiotemporal feature sequences of the video to be detected. Spatial and temporal features can be extracted separately through spatial and temporal Transformers, and then combined for detection to ensure detection accuracy. Finally, a classifier aggregates the spatiotemporal feature sequences into decision vectors and performs label prediction to obtain the corresponding forgery probability. This allows for further processing of the extracted spatiotemporal features to obtain the desired output type, which meets practical needs.

[0041] In some embodiments of this application, the step of extracting features from each video frame of each video segment, wherein the extraction is performed by obtaining a multidimensional feature map of the video frame based on depthwise separable convolution, and video frames belonging to the same video segment are aggregated to obtain an intra-frame feature sequence for each video segment, specifically includes: Based on the multiple depthwise separable convolutional modules of the feature extraction module, feature extraction is performed on each video frame of each video segment to obtain a multidimensional feature map of each video frame of each video segment; By stretching the dimensions, the multidimensional feature map of each video frame of each video segment is converted into a corresponding one-dimensional vector. The one-dimensional vectors of video frames belonging to the same video segment are then aggregated. During aggregation, the vectors are arranged according to the timestamp order of the video frames to obtain the intra-frame feature sequence of each video segment.

[0042] Specifically, the preferred implementation of the feature extraction module architecture is the shallow network portion of the Xception network architecture, used to capture structural features such as texture, edges, and facial organs related to forgery traces in face images. More specifically, when performing feature extraction through multiple depthwise separable convolutional modules, depthwise convolution is first performed on each channel individually to preserve local features within the channel; then, pointwise convolution (with a kernel size of 1×1) is used to achieve feature fusion between different channels. Compared to standard convolution operations, this reduces the number of parameters by approximately 70% while fully preserving key facial feature information. After feature fusion processing, each video frame outputs a multidimensional feature map with "spatial dimension + channel dimension". The dimensions of the multidimensional feature map are derived from the network architecture design and the size of the input video frame. For example, the dimension of a certain multidimensional feature map is represented as 28×28×512, where 28×28 is the spatial dimension size and 512 is the number of channels.

[0043] Specifically, multidimensional feature maps are converted into corresponding one-dimensional vectors by dimensional stretching. That is, multidimensional feature maps are flattened in a certain dimensional order. For example, a feature map with dimensions of 28×28×512 should be flattened to obtain a one-dimensional vector with a length of 36128. Dimension stretching can reduce the dimensionality from multidimensional features to one-dimensional features without losing any feature information during the dimensionality reduction process.

[0044] Specifically, since four video segments are obtained when sampling the video to be detected, and each video segment consists of eight consecutive video frames, each video frame of each video segment can be flattened into a one-dimensional vector with a length of 36128 (dimension 1×36128). After the flattening operation of dimensional stretching, within each video segment, the video frames are arranged and stacked in rows according to the timestamp order of each video frame to achieve the aggregation operation, thereby obtaining the intra-frame feature sequence of each video segment. The dimension of this intra-frame feature sequence is n×d, where n=8, representing the number of video frames contained in each video segment, and d=36128, representing the vector length of each video frame after flattening. Through dimensional stretching and aggregation, the feature information of each video frame is preserved, and the temporal order relationship between frames is maintained, providing a suitable input format for subsequent temporal feature extraction.

[0045] This application first performs feature extraction based on multiple depthwise separable convolutional modules to obtain multidimensional feature maps for each video frame of each video segment. This allows for multidimensional feature extraction through multiple depthwise separable convolutional modules, fully recognizing and acquiring the features of the input frame. Then, the multidimensional feature maps are converted into corresponding one-dimensional vectors through dimensional stretching and aggregated. During aggregation, the vectors are arranged according to the timestamp order of the video frames to obtain the intra-frame feature sequence of each video segment. This approach can maintain the multidimensional features recognized by depthwise separable convolution while preserving the format of the intra-frame feature sequence through the timestamp order, providing an adapted input format for subsequent temporal feature extraction.

[0046] In some embodiments of this application, the transformation of the query sequence and value sequence calculated from the intra-frame feature sequence by the attention module of the current coding layer into attention output specifically includes: Linear projection is performed on the intra-frame feature sequence to obtain the corresponding query sequence and value sequence; Global projection is performed on each query token of the query sequence and each value token of the value sequence to obtain the importance score of each query token and the importance score of each value token. Based on the importance score of each query token, the query sequence is weighted and summed to obtain the query global context vector corresponding to the query sequence. Based on the importance score of each value token, the value sequence is weighted and summed to obtain the value global context vector corresponding to the value sequence. The query global context vector is fused with the value sequence to obtain a first fusion matrix, the value global context vector is fused with the query sequence to obtain a second fusion matrix, and the first fusion matrix and the second fusion matrix are integrated to obtain the attention output.

[0047] For a detailed explanation of the implementation logic of the attention module in each coding layer of this application, please refer to [link to relevant documentation]. Figure 2 This application provides an attention module based on a symmetric global weighted attention mechanism. In this attention module, the query sequence and value sequence calculated from the intra-frame feature sequence are transformed into attention output, including steps S201 to S204, each of which is as follows: Step S201: Perform linear projection on the input data to obtain the query sequence. Sum sequence .

[0048] Specifically, the input data has a dimension of n×d; when performing a linear projection on the input data, it is necessary to use a corresponding projection matrix to query the sequence. The corresponding projection matrix is value sequence The corresponding projection matrix is Projection matrix The dimensions of all elements are d×d; linear projection is performed on the input data, which involves multiplying the input data with the projection matrix. The operation, at this point, yields the query sequence obtained through projection. Sum sequence The dimension remains d×d.

[0049] Step S202: For the query sequence Each query token is globally projected to obtain an importance score for that query token. ; value sequence Each value token is globally projected to obtain an importance score for each value token. .

[0050] Specifically, query sequence Each query token represents a query sequence. The query row vector consisting of n rows from each row in the array. Each query row vector has a dimension of 1×d; when performing global projection on the query token, a global projection vector is needed. Its dimension is 1×d; perform a global projection on the query token, that is, project each query row vector With global projection vector Perform a vector dot product operation to obtain the corresponding scalar value, and then activate the resulting scalar using the Sigmoid activation function. The activation result is then the vector for each query row. Importance score , that is .

[0051] Similarly, value sequence Each value token represents a sequence of values. The n value row vectors formed by each row in the middle Each value row vector has a dimension of 1×d; when performing a global projection on the value Token, a global projection vector is needed. Its dimension is 1×d; perform a global projection on the value Token, that is, the row vector of each value. With global projection vector Perform a vector dot product operation to obtain the corresponding scalar value, and then activate the resulting scalar using the Sigmoid activation function. The resulting activated scalar is then a row vector for each value. Importance score , that is .

[0052] Step S203: Based on the importance score of each query token For the query sequence Perform a weighted summation to obtain the query global context vector. ;Score based on the importance of each token value , value sequence Perform a weighted summation to obtain the global context vector. .

[0053] Specifically, based on the importance score of each query token. For the query sequence Perform a weighted summation, that is, sum the vectors of each query row. Corresponding importance score Perform vector scalar multiplication (element-by-element multiplication). Operation (Importance Score) That is, query row vector (weights), and each query row vector Scalar multiplication results are used for vector summation. The summation result at this point is the query sequence. Corresponding query global context vector Its dimension is 1×d.

[0054] Similarly, based on the importance score of each value token. , value sequence Perform a weighted summation, that is, sum the values ​​in each row vector. Corresponding importance score Perform vector scalar multiplication (element-by-element multiplication). Operation (Importance Score) That is, query row vector (weights), and row vector each value Scalar multiplication results are used for vector summation. The summation result at this point is the value sequence. The corresponding value is the global context vector. Its dimension is 1×d.

[0055] Step S204: Query the global context vector AND value sequence Perform fusion to obtain the first fusion matrix; then convert the global context vector. With query sequence The first and second fusion matrices are fused to obtain the second fusion matrix; the attention output is obtained by integrating the first and second fusion matrices.

[0056] Specifically, the query global context vector will be used. AND value sequence The fusion process yields the first fusion matrix, which is achieved by first broadcasting the query global context vector of dimension 1×d. Perform dimensional expansion to an n×d dimension, and then broadcast the query global context vector. AND value sequence Perform element-wise multiplication The operation yields the first fusion matrix, which has dimensions n×d.

[0057] Similarly, the global context vector is valued. With query sequence The fusion process yields a second fusion matrix, which is achieved by first broadcasting the values ​​of the 1×d-dimensional global context vector. Perform dimensional expansion to an n×d dimension, then broadcast the values ​​to the global context vector. With query sequence Perform element-wise multiplication The operation yields a second fusion matrix with dimensions n×d.

[0058] Specifically, the attention output is obtained by integrating the first fusion matrix and the second fusion matrix, that is, by adding the first fusion matrix and the second fusion matrix together. The operation, the sum of which is the corresponding attention output, has a dimension of n×d.

[0059] This application first performs linear projection on the intra-frame feature sequence to obtain the query sequence and value sequence. Then, it performs global projection on the corresponding query token and value token to obtain the importance score of each query token and value token. Subsequently, it performs weighted summation on the query sequence and value sequence to obtain the corresponding global context vector. By performing corresponding projection transformation on the query sequence and value sequence through the global context vector, it can replace the role of the key sequence in the existing standard attention mechanism, reduce the computational dimension and the number of computational parameters, thereby reducing computational complexity and improving computational efficiency. Finally, it performs symmetric cross-fusion on the query sequence, value sequence and the corresponding global context vector, which can fully fuse to generate the corresponding attention and maintain the corresponding detection accuracy.

[0060] In some embodiments of this application, the step of calibrating the attention output based on the feedforward network module of the current coding layer to obtain the intra-frame feature sequence for the next iteration specifically includes: The attention output is subjected to a higher-dimensional linear mapping and activated by an activation function with leakage to obtain a first intermediate feature. The first intermediate feature is then subjected to global average pooling to obtain a second intermediate feature. The second intermediate feature is sequentially subjected to a dimensionality reduction linear mapping and a dimensionality increase linear mapping to extract channel weights, resulting in a channel weight vector; wherein the dimension of the channel weight vector is the same as that of the second intermediate feature. The channel weight vector is fused with the first intermediate feature to obtain the intra-frame feature sequence for the next iteration.

[0061] For a detailed explanation of the implementation logic of the feedforward network module in each coding layer of this application, please refer to [link to relevant documentation]. Figure 3 The feedforward neural network module provided in this application calibrates the attention output to obtain the starting data for the next iteration, including steps S301 to S303, each step of which is as follows: Step S301: Perform a linear mapping on the attention output and activate it using an activation function with leakage to obtain the first intermediate feature; perform global average pooling on the first intermediate feature to obtain the second intermediate feature.

[0062] Specifically, the attention output is subjected to a higher-dimensional linear mapping ( Figure 3 When using a Linear layer between an n×d matrix and an n×2d matrix on the input side, the dimension is expanded from n×d to n×2d (expansion coefficient of 2). During expansion, the weights are initialized using He normality, and the bias is initialized to 0. Activation is applied to the dimensionality-upgraded result (…). Figure 3 When using ReLU layers between n×d and n×2d matrices, activation is performed using a leaky ReLU function with a leakage coefficient of 0.01. Activation with a leaky activation function can avoid the problem of neuron death in the neural network and keep the parameters of the neural network updated during training.

[0063] Specifically, after obtaining the first intermediate feature (with dimensions n×2d), global average pooling is performed on the first intermediate feature according to the channel dimension, compressing the n spatial elements of each channel into a scalar value to obtain the second intermediate feature. Figure 3 The global average pooling operation is not presented as a layer in the paper, but has a dimension of 1×2d. Through global average pooling, channel-level feature descriptions can be captured, so channel weights can be further extracted and high-significance channels can be enhanced and low-significance channels can be suppressed based on the channel weights.

[0064] Step S302: Perform dimensionality reduction linear mapping and dimensionality increase linear mapping on the second intermediate feature in sequence to extract channel weights and obtain channel weight vectors; wherein the dimension of the channel weight vector is the same as that of the second intermediate feature.

[0065] Specifically, through a two-layer fully connected network ( Figure 3The two linear layers in the Linear-Relu-Linear-Sigmoid hierarchical structure first perform a dimension reduction linear mapping on the second intermediate feature, and then perform a dimension increase linear mapping. During the dimension reduction linear mapping, the second intermediate feature, which is used to represent the channel feature with a dimension of 1×2d, is reduced to a dimension of 1×(2d / 16), and activated by the ReLU activation function. Figure 3 The ReLU layer in the Linear-ReLU-Linear-Sigmoid hierarchical structure); when performing the upscaling linear mapping, the features of dimension 1×(2d / 16) are restored to dimension 1×2d and activated by the Sigmoid activation function ( Figure 3 The corresponding channel weight vector is obtained by first increasing the dimensionality of the second intermediate feature that represents the channel features and then decreasing the dimensionality. The channel weight vector used to indicate the weight of each channel in the second intermediate feature can be accurately extracted.

[0066] Step S303: Fuse the channel weight vector with the first intermediate feature to obtain the initial data for the next iteration.

[0067] Specifically, the channel weight vector has a dimension of 1×2d, and the first intermediate feature has a dimension of n×2d; when fusing the channel weight vector and the first intermediate feature, the channel weight vector and the first intermediate feature are first multiplied element-wise channel by channel. The operation involves multiplying each of the n elements of each channel in the first intermediate feature by the corresponding channel weight in the channel weight vector, thereby enhancing highly significant channels and suppressing less significant channels based on their channel weights; then, the result of the channel-by-channel, element-by-element multiplication is subjected to a dimension reduction linear mapping (…). Figure 3 The Linear layer preceding the n×d matrix on the output side reduces the dimension from n×2d to n×d, obtaining the initial data for the next iteration, which is the input to the encoding layer in the next iteration.

[0068] This application first performs a dimensionality-upgrading linear mapping on the attention output and activates it using an activation function with leakage. Then, global average pooling is performed to obtain the second intermediate feature. Next, dimensionality-downgrading and dimensionality-upgrading linear mappings are performed sequentially to extract channel weights. Finally, the channel weights and the first intermediate feature are fused to obtain the intra-frame feature sequence. By extracting channel weights and fusing them with the corresponding features, channels with high channel weights in the features can be enhanced, while channels with low channel weights can be suppressed, thereby improving feature saliency and maintaining the corresponding detection accuracy.

[0069] In some embodiments of this application, the processing logic and specific data flow of the temporal Transformer module of the first detection model are the same as those of the spatial Transformer module, and therefore will not be repeated here. It is easy to understand that, given that the original data consists of four video segments, each containing eight consecutive video frames, in the spatial Transformer module, since the input data is the intra-frame feature sequence of each video segment, which is actually obtained by processing eight consecutive video frames, the spatial Transformer module focuses more on processing the differences in the "continuous" process within the video segment, i.e., increasing attention to forgery traces in the spatial dimension of the face image, including facial detail tampering, texture inconsistencies, etc.; while in the temporal Transformer module, since the input data is the inter-frame feature sequence of the video to be detected, which is actually obtained by processing four video segments, the temporal Transformer module focuses more on processing the differences between different video segments or between frames, i.e., increasing attention to forgery traces in the temporal dimension of the face image, including sudden changes in facial expression, incoherent actions between frames, etc. Although the spatial Transformer module and the temporal Transformer module have the same data flow and processing logic, by constructing and dividing different data dimensions and data formats, it is possible to force the spatial Transformer module and the temporal Transformer module to focus on different types of dimensions for processing, so as to extract the spatial features and temporal features of the video to be detected respectively.

[0070] In some embodiments of this application, the step of aggregating the spatiotemporal feature sequence into a decision vector, and performing label prediction based on the decision vector to obtain and output the forgery probability of the video to be detected specifically includes: Global pooling is performed on the spatiotemporal feature sequence to aggregate the feature information in the spatiotemporal feature sequence and obtain a decision vector; Based on the fully connected layer of the classifier, label prediction is performed on the decision vector to obtain and output the forgery probability of the video to be detected.

[0071] This application first performs global pooling on the spatiotemporal feature sequence to obtain the decision vector, and then predicts the corresponding forgery probability based on the fully connected layer. It can aggregate the feature information of the spatiotemporal feature sequence and further process it into the expected output type, which meets the actual needs.

[0072] Step S103: Obtain the detection result of the video to be detected based on the forgery probability.

[0073] Specifically, when the probability of forgery is greater than the preset forgery threshold, the video to be detected is determined to be a forged video; otherwise, it is determined to be a real video. The preferred value for the preset forgery threshold is 0.5.

[0074] Generally, the data flow during training of the first detection model in this application is the same as the data flow during detection given in this application, so the data flow during training of the first detection model and the detailed training process will not be described again here.

[0075] Specifically, during training, the first detection model uses binary cross-entropy loss to calculate the error between the model's predicted value and the sample's true label. The formula for calculating binary cross-entropy loss is as follows: ; in, This represents the total number of video samples in the current batch. For the first The true label of each sample To fake the video, This is a real video; For the first The probability of forgery predicted for each sample; To balance the weights for each category, , To increase the number of real video samples in the training set, The number of fake video samples in the training set is determined by balancing the weights by category. This can prevent the model from being biased towards predicting a particular class due to an excessive number of samples of that class during training.

[0076] Specifically, during the training process, the first detection model uses the backpropagation algorithm to propagate the loss value back to each layer of the model network to update all learnable parameters. The preferred implementation of the backpropagation algorithm is the Adam optimizer. During the training process, the first detection model repeats the training process of "input data - loss calculation - parameter update" until the loss value converges. If the loss value does not decrease significantly for 10 consecutive epochs, it is determined to have converged and training is stopped, thus obtaining the first detection model that has been trained.

[0077] This application first samples the video to be detected to obtain multiple video segments, and each video segment contains multiple consecutive video frames, ensuring comprehensive sampling and providing data support for maintaining detection accuracy. Then, the multiple video segments are input into a first detection model for detection. The first detection model is configured with spatial Transformer modules and temporal Transformer modules, each containing multiple identical coding layers. Each coding layer is configured with an attention module and a feedforward network module. The attention module is based on a symmetric global weighted attention mechanism, transforming the query sequence and value sequence calculated from the input data into attention outputs. The first detection model outputs the forgery probability of the video to be detected, and the detection result of the video to be detected is obtained based on the forgery probability. By setting spatial and temporal Transformers, the spatial and temporal features of the video to be detected can be extracted respectively, and the combination of the two can ensure detection accuracy. By specially configuring the attention module of the coding layer, based on the symmetric global weighted attention mechanism, the query sequence and value sequence are directly transformed without calculating the key sequence in the existing standard attention mechanism, which can reduce the computational dimension and the number of computational parameters, thereby reducing computational complexity and improving computational efficiency, thus improving the detection efficiency of forged faces.

[0078] For a method corresponding to the one described above, please refer to [link to relevant documentation]. Figure 4 This application provides a fake face video detection device, including a video sampling module 410, a model detection module 420 and a result output module 430; The video sampling module 410 is used to sample the video to be detected to obtain multiple video segments; wherein each video segment contains multiple consecutive video frames. The model detection module 420 is used to input the multiple video segments into a preset first detection model for detection and output the forgery probability of the video to be detected; wherein, the first detection model is configured with a spatial Transformer module and a temporal Transformer module in sequence; both the spatial Transformer module and the temporal Transformer module contain multiple identical encoding layers; the encoding layers are configured with an attention module and a feedforward network module in sequence; the attention module is based on a symmetric global weighted attention mechanism, which transforms the query sequence and value sequence calculated from the input data into attention output; The result output module 430 is used to obtain the detection result of the video to be detected based on the forgery probability.

[0079] In some embodiments of this application, the video sampling module 410 includes a random sampling unit and a segment processing unit; The random sampling unit is used to randomly sample the video to be detected to obtain multiple original video segments; wherein each original video segment consists of multiple consecutive original video frames. The segment processing unit is used to detect the face region of each original video frame in each original video segment, crop the face region, and perform an affine transformation to align the face key points of each original video frame in each original video segment, thereby obtaining multiple video segments.

[0080] In some embodiments of this application, the first detection model is further provided with a feature extraction module before the spatial Transformer module and a classifier after the temporal Transformer module; the model detection module 420 includes a feature extraction unit, a spatial processing unit, a temporal processing unit and a classification prediction unit; The feature extraction unit is used to extract features from each video frame of each video segment through the feature extraction module. During extraction, a multi-dimensional feature map of the video frame is obtained based on depthwise separable convolution, and video frames belonging to the same video segment are aggregated to obtain the intra-frame feature sequence of each video segment. The spatial processing unit is used to iteratively process the intra-frame feature sequence of each video segment based on the multiple coding layers through the spatial Transformer module. During iteration, the query sequence and value sequence calculated from the intra-frame feature sequence are transformed into attention output according to the attention module of the current coding layer, and the attention output is calibrated according to the feedforward network module of the current coding layer to obtain the intra-frame feature sequence for the next iteration, until each coding layer of the spatial Transformer module has been processed to obtain the spatial feature sequence of each video segment, and the spatial feature sequences of each video segment are aggregated to obtain the inter-frame feature sequence of the video to be detected. The temporal processing unit is used to iteratively process the inter-frame feature sequence based on the multiple coding layers through the temporal Transformer module. During iteration, the query sequence and value sequence calculated by the inter-frame feature sequence are transformed into attention output according to the attention module of the current coding layer, and the attention output is calibrated according to the feedforward network module of the current coding layer to obtain the inter-frame feature sequence for the next iteration, until each coding layer of the temporal Transformer module has been processed to obtain the spatiotemporal feature sequence of the video to be detected. The classification and prediction unit is used to aggregate the spatiotemporal feature sequence into a decision vector through the classifier, and perform label prediction based on the decision vector to obtain and output the forgery probability of the video to be detected.

[0081] In some embodiments of this application, the step of extracting features from each video frame of each video segment, wherein the extraction is performed by obtaining a multidimensional feature map of the video frame based on depthwise separable convolution, and video frames belonging to the same video segment are aggregated to obtain an intra-frame feature sequence for each video segment, specifically includes: Based on the multiple depthwise separable convolutional modules of the feature extraction module, feature extraction is performed on each video frame of each video segment to obtain a multidimensional feature map of each video frame of each video segment; By stretching the dimensions, the multidimensional feature map of each video frame of each video segment is converted into a corresponding one-dimensional vector. The one-dimensional vectors of video frames belonging to the same video segment are then aggregated. During aggregation, the vectors are arranged according to the timestamp order of the video frames to obtain the intra-frame feature sequence of each video segment.

[0082] In some embodiments of this application, the transformation of the query sequence and value sequence calculated from the intra-frame feature sequence by the attention module of the current coding layer into attention output specifically includes: Linear projection is performed on the intra-frame feature sequence to obtain the corresponding query sequence and value sequence; Global projection is performed on each query token of the query sequence and each value token of the value sequence to obtain the importance score of each query token and the importance score of each value token. Based on the importance score of each query token, the query sequence is weighted and summed to obtain the query global context vector corresponding to the query sequence. Based on the importance score of each value token, the value sequence is weighted and summed to obtain the value global context vector corresponding to the value sequence. The query global context vector is fused with the value sequence to obtain a first fusion matrix. The value global context vector is fused with the query sequence to obtain a second fusion matrix. The first fusion matrix and the second fusion matrix are then integrated to obtain the attention output.

[0083] In some embodiments of this application, the step of calibrating the attention output based on the feedforward network module of the current coding layer to obtain the intra-frame feature sequence for the next iteration specifically includes: The attention output is subjected to a higher-dimensional linear mapping and activated by an activation function with leakage to obtain a first intermediate feature. The first intermediate feature is then subjected to global average pooling to obtain a second intermediate feature. The second intermediate feature is sequentially subjected to a dimensionality reduction linear mapping and a dimensionality increase linear mapping to extract channel weights, resulting in a channel weight vector; wherein the dimension of the channel weight vector is the same as that of the second intermediate feature. The channel weight vector is fused with the first intermediate feature to obtain the intra-frame feature sequence for the next iteration.

[0084] In some embodiments of this application, the step of aggregating the spatiotemporal feature sequence into a decision vector, and performing label prediction based on the decision vector to obtain and output the forgery probability of the video to be detected specifically includes: Global pooling is performed on the spatiotemporal feature sequence to aggregate the feature information in the spatiotemporal feature sequence and obtain a decision vector; Based on the fully connected layer of the classifier, label prediction is performed on the decision vector to obtain and output the forgery probability of the video to be detected.

[0085] This application first samples the video to be detected to obtain multiple video segments, and each video segment contains multiple consecutive video frames, ensuring comprehensive sampling and providing data support for maintaining detection accuracy. Then, the multiple video segments are input into a first detection model for detection. The first detection model is configured with spatial Transformer modules and temporal Transformer modules, each containing multiple identical coding layers. Each coding layer is configured with an attention module and a feedforward network module. The attention module is based on a symmetric global weighted attention mechanism, transforming the query sequence and value sequence calculated from the input data into attention outputs. The first detection model outputs the forgery probability of the video to be detected, and the detection result of the video to be detected is obtained based on the forgery probability. By setting spatial and temporal Transformers, the spatial and temporal features of the video to be detected can be extracted respectively, and the combination of the two can ensure detection accuracy. By specially configuring the attention module of the coding layer, based on the symmetric global weighted attention mechanism, the query sequence and value sequence are directly transformed without calculating the key sequence in the existing standard attention mechanism, which can reduce the computational dimension and the number of computational parameters, thereby reducing computational complexity and improving computational efficiency, thus improving the detection efficiency of forged faces.

[0086] It should be understood that the apparatus provided in this application is corresponding to the aforementioned method. The fake face video detection apparatus provided in this application can implement the fake face video detection method provided in any of the embodiments of this application.

[0087] Adaptively, embodiments of this application also provide a computer device and a computer-readable storage medium.

[0088] The computer device includes: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor; The processor executes the computer program to implement a method for detecting fake face videos according to this application.

[0089] The computer-readable storage medium stores multiple instructions adapted for loading by a processor to execute a method for detecting fake face videos according to this application.

[0090] The above description represents some embodiments of this application, providing a further detailed explanation of the purpose, technical solution, and beneficial effects of this application. It should be understood that the above-described embodiments of this application should not be construed as limiting this application. In particular, any changes, modifications, equivalent substitutions, and variations made by those skilled in the art within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for detecting forged face videos, characterized in that, include: The video to be tested is sampled to obtain multiple video segments; each video segment contains multiple consecutive video frames. The multiple video segments are input into a preset first detection model for detection, and the forgery probability of the video to be detected is output. The first detection model sequentially includes a spatial Transformer module and a temporal Transformer module. Both the spatial and temporal Transformer modules contain multiple identical encoding layers. Each encoding layer sequentially includes an attention module and a feedforward network module. The attention module, based on a symmetric global weighted attention mechanism, transforms the query sequence and value sequence calculated from the input data into an attention output. The detection result of the video to be detected is obtained based on the forgery probability.

2. The method for detecting fake face videos according to claim 1, characterized in that, The process of sampling the video to be detected yields multiple video segments, specifically including: The video to be detected is randomly sampled to obtain multiple original video segments; wherein each original video segment consists of multiple consecutive original video frames; The face region of each original video frame in each original video segment is detected, and the face region is cropped and subjected to affine transformation to align the face key points of each original video frame in each original video segment, resulting in multiple video segments.

3. The method for detecting fake face videos according to claim 1, characterized in that, The first detection model includes a feature extraction module before the spatial Transformer module and a classifier after the temporal Transformer module. The process of inputting the multiple video segments into the preset first detection model for detection and outputting the forgery probability of the video to be detected specifically includes: The feature extraction module extracts features from each video frame of each video segment. During extraction, a multi-dimensional feature map of the video frame is obtained based on depthwise separable convolution, and video frames belonging to the same video segment are aggregated to obtain the intra-frame feature sequence of each video segment. The spatial Transformer module iteratively processes the intra-frame feature sequence of each video segment based on the multiple coding layers. During iteration, the query sequence and value sequence calculated from the intra-frame feature sequence are transformed into attention output according to the attention module of the current coding layer. The attention output is then calibrated according to the feedforward network module of the current coding layer to obtain the intra-frame feature sequence for the next iteration. This process continues until each coding layer of the spatial Transformer module has been processed, resulting in the spatial feature sequence of each video segment. The spatial feature sequences of each video segment are then aggregated to obtain the inter-frame feature sequence of the video to be detected. The temporal Transformer module iteratively processes the inter-frame feature sequence based on the multiple coding layers. During iteration, the query sequence and value sequence calculated from the inter-frame feature sequence are transformed into attention outputs according to the attention module of the current coding layer. The attention outputs are then calibrated according to the feedforward network module of the current coding layer to obtain the inter-frame feature sequence for the next iteration. This process continues until each coding layer of the temporal Transformer module has been processed, resulting in the spatiotemporal feature sequence of the video to be detected. The classifier aggregates the spatiotemporal feature sequences into a decision vector, and performs label prediction based on the decision vector to obtain and output the forgery probability of the video to be detected.

4. The method for detecting fake face videos according to claim 3, characterized in that, The process involves feature extraction for each video frame of each video segment. During extraction, a multi-dimensional feature map of the video frame is obtained based on depthwise separable convolution. Video frames belonging to the same video segment are then aggregated to obtain an intra-frame feature sequence for each video segment. Specifically, this includes: Based on the multiple depthwise separable convolutional modules of the feature extraction module, feature extraction is performed on each video frame of each video segment to obtain a multidimensional feature map of each video frame of each video segment; By stretching the dimensions, the multidimensional feature map of each video frame of each video segment is converted into a corresponding one-dimensional vector. The one-dimensional vectors of video frames belonging to the same video segment are then aggregated. During aggregation, the vectors are arranged according to the timestamp order of the video frames to obtain the intra-frame feature sequence of each video segment.

5. The method for detecting fake face videos according to claim 3, characterized in that, The process of transforming the query sequence and value sequence calculated from the intra-frame feature sequence into attention output based on the attention module of the current coding layer specifically includes: Linear projection is performed on the intra-frame feature sequence to obtain the corresponding query sequence and value sequence; Global projection is performed on each query token of the query sequence and each value token of the value sequence to obtain the importance score of each query token and the importance score of each value token. Based on the importance score of each query token, the query sequence is weighted and summed to obtain the query global context vector corresponding to the query sequence. Based on the importance score of each value token, the value sequence is weighted and summed to obtain the value global context vector corresponding to the value sequence. The query global context vector is fused with the value sequence to obtain a first fusion matrix, the value global context vector is fused with the query sequence to obtain a second fusion matrix, and the first fusion matrix and the second fusion matrix are integrated to obtain the attention output.

6. The method for detecting fake face videos according to claim 3, characterized in that, The step of calibrating the attention output based on the current coding layer's feedforward network module to obtain the intra-frame feature sequence for the next iteration specifically includes: The attention output is subjected to a higher-dimensional linear mapping and activated by an activation function with leakage to obtain a first intermediate feature. The first intermediate feature is then subjected to global average pooling to obtain a second intermediate feature. The second intermediate feature is sequentially subjected to a dimensionality reduction linear mapping and a dimensionality increase linear mapping to extract channel weights, resulting in a channel weight vector; wherein the dimension of the channel weight vector is the same as that of the second intermediate feature. The channel weight vector is fused with the first intermediate feature to obtain the intra-frame feature sequence for the next iteration.

7. The method for detecting fake face videos according to claim 3, characterized in that, The step of aggregating the spatiotemporal feature sequences into a decision vector, and performing label prediction based on the decision vector to obtain and output the forgery probability of the video to be detected specifically includes: Global pooling is performed on the spatiotemporal feature sequence to aggregate the feature information in the spatiotemporal feature sequence and obtain a decision vector; Based on the fully connected layer of the classifier, label prediction is performed on the decision vector to obtain and output the forgery probability of the video to be detected.

8. A device for detecting fake face videos, characterized in that, It includes a video sampling module, a model detection module, and a result output module; The video sampling module is used to sample the video to be detected to obtain multiple video segments; wherein each video segment contains multiple consecutive video frames; The model detection module is used to input the multiple video segments into a preset first detection model for detection and output the forgery probability of the video to be detected. The first detection model sequentially includes a spatial Transformer module and a temporal Transformer module. Both the spatial and temporal Transformer modules contain multiple identical encoding layers. Each encoding layer sequentially includes an attention module and a feedforward network module. The attention module, based on a symmetric global weighted attention mechanism, transforms the query sequence and value sequence calculated from the input data into an attention output. The result output module is used to obtain the detection result of the video to be detected based on the forgery probability.

9. A computer device, characterized in that, include: processor; Memory; A computer program stored in the memory and configured to be executed by the processor; The processor, when executing the computer program, implements a method for detecting fake face videos as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute a method for detecting fake face videos as described in any one of claims 1 to 7.

Images