Method, system, device, processor and computer readable storage medium thereof for realizing deep fake face identification

Abstract

This invention relates to a method for identifying deepfake faces based on rPPG multi-scale spatiotemporal graphs and a two-stage model. The method includes: (1) collecting a deepfake face video dataset and preprocessing the video; (2) generating rPPG multi-scale spatiotemporal graphs; (3) constructing a mask-guided local attention module for the first stage of training, extracting features from a single rPPG spatiotemporal graph; (4) constructing a Transformer-based temporal aggregation module for the second stage of training, fusing the combined features of multiple adjacent spatiotemporal graphs; and (6) constructing a classification head for classification and recognition, and constructing a loss function. This invention also relates to a corresponding system, device, processor, and storage medium. Using the method, system, device, processor, and storage medium of this invention, the combined features of multiple spatiotemporal graphs representing a video are extracted through a two-stage model, resulting in better interpretability and fake face identification performance compared to baseline models.

Description

Technical Field

[0001] This invention relates to the field of digital image technology, and more particularly to the field of computer vision technology. Specifically, it relates to a method, system, device, processor, and computer-readable storage medium for identifying deepfake faces based on rPPG multi-scale spatiotemporal graphs and a two-stage model. Background Technology

[0002] With the development of generative deep learning models, the technical barrier to deepfake face creation is decreasing, allowing individuals to easily create realistic fake faces using publicly available models or tools. Deepfakes can also be abused by malicious users to create false political information or spread pornography. As a defense mechanism, face spoofing identification technologies have been developed to mitigate the risks posed by deepfakes. Remote photoplethysmography (rPPG) extracts heartbeat signals from recorded videos by examining subtle changes in skin color caused by cardiac activity. Since face spoofing inevitably disrupts the periodic changes in facial color, rPPG has proven to be a biometric signal that can be effectively used to identify fake faces.

[0003] However, most existing deep face forgery detection methods based on rPPG signals still have some shortcomings. For example, the invention patent application with application number CN202210572034.2 extracts heart rate signals from 32 small square boxes on the face in each frame, but these ROI regions overlap and are of a single scale; moreover, it only uses a one-stage encoder to extract features from a single rPPG spatiotemporal map, without considering feature fusion of multiple adjacent rPPG spatiotemporal maps; and it only uses binary cross-entropy loss, without considering attention weights at pixel-level local locations, resulting in limited detection performance. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, system, device, processor and computer-readable storage medium for deepfake face identification based on rPPG multi-scale spatiotemporal graph, which can effectively consider the comprehensive features of multiple adjacent video segments.

[0005] To achieve the above objectives, the present invention provides a method, system, apparatus, processor, and computer-readable storage medium for deepfake face identification based on rPPG multi-scale spatiotemporal graphs and a two-stage model, as follows:

[0006] The method for identifying deepfake faces based on rPPG multi-scale spatiotemporal graphs and a two-stage model is characterized by the following steps:

[0007] (1) Collect a dataset of deepfake face videos and perform preprocessing operations on the video data to obtain a set of cropped face video frames;

[0008] (2) Generate an rPPG multi-scale spatiotemporal map based on the cropped face video frames;

[0009] (3) Construct a mask-guided local attention module, perform the first stage of training, and extract features of a single rPPG spatiotemporal graph;

[0010] (4) Construct the Transformer module, conduct the second stage training, and integrate the comprehensive features of multiple adjacent rPPG spatiotemporal graphs;

[0011] (5) Construct a classification head, pool the fused high-dimensional features and perform classification and recognition processing to obtain the identification results of the target image and construct the overall loss function.

[0012] Preferably, step (2) specifically includes the following steps:

[0013] (2.1) Divide a complete video into several T-frame segments with a step size of ω frames;

[0014] (2.2) For each frame, perform face alignment and extract facial key points;

[0015] (2.3) Based on the aforementioned facial key points, select n heartbeat signal information regions to form a set of Regions of Interest (ROIs) R. t ={R 1t ,R 2t ,…,R nt};

[0016] (2.4) For the aforementioned ROI set R t For all non-empty subsets, calculate the mean of all pixels contained in each non-empty subset to obtain 2. n -1 pixel average across RGB three channels;

[0017] (2.5) For each video segment, the operations in steps (2.2) to (2.4) are performed on its T frames to obtain a result with dimension T×(2). n -1)×3 multi-scale spatiotemporal diagram, where T is the time length, 2 n -1 represents the number of different information area combinations, and 3 represents the number of RGB channels.

[0018] Preferably, the n information regions in (2.3) are the forehead, chin, upper left and right cheeks, and lower left and right cheeks, as shown in the specific regions. Figure 2 As shown.

[0019] Preferably, step (3) specifically includes the following steps:

[0020] (3.1) Construct EfficientNet as the backbone network f(·) for the input rPPG spatiotemporal graph. Features are extracted through the backbone network, resulting in the mid-layer feature map F. m =f mid (X)∈R C×H×W Where C, H, and W represent the number of channels, columns, and rows of the feature map, respectively;

[0021] (3.2) Construct a mask-guided local attention module, using the mid-layer feature map F m As input, we obtain the attention mask A. mask :

[0022] A mask =Sigmoid(Conv(F m ))

[0023] Where Conv(·) represents the convolution operation;

[0024] (3.3) Combine the attention mask with the middle layer feature map F m Perform a dot product operation to obtain the position-weighted feature map F′=A mask ·F m And use F′ as input for feature extraction in subsequent network layers;

[0025] (3.4) Calculate the pixel-level mask label A of the aforementioned rPPG spatiotemporal graph. gt For rPPG spatiotemporal graphs generated from fake videos, find their corresponding real rPPG spatiotemporal graphs, perform pixel-by-pixel subtraction to obtain residual spatiotemporal graphs, then convert the residual spatiotemporal graphs to grayscale, normalize them to 0 to 1, and adjust their size to match the attention mask A. mask For masks of the same size, binarization with a threshold of 0.1 yields the corresponding pixel-level mask label A. gt ;

[0026] (3.5) Apply the attention mask A mask and the corresponding pixel-level mask label A gt Calculate the L1 distance using the following formula, and use it as the mask loss function L. mask :

[0027]

[0028] More preferably, step (4) specifically includes the following steps:

[0029] (4.1) Input the K temporally adjacent rPPG spatiotemporal maps into the backbone network trained in the first stage to obtain K global high-dimensional features F. h Then, global average pooling is performed, and classification encoding and one-dimensional learnable positional encoding are superimposed as the input sequence Z of the Transformer. in ;

[0030] (4.2) Construct a feature fusion module based on Transformer for multiple rPPG spatiotemporal graphs: The input sequence Z... in Multi-head self-attention (MSA) operations are performed, followed by a feedforward network (FFN). After each operation, layer normalization (LN) and residual connections are used to further adjust the output, resulting in the Transformer output Z. out .

[0031] More preferably, step (4.2) specifically includes the following steps:

[0032] (4.2.1) Input sequence Z in A query matrix is ​​generated through a linear mapping layer. A key matrix and a Value matrix Next, the three matrices are passed into the multi-head self-attention mechanism MSA, as shown in the following formula:

[0033]

[0034] Where d is the normalization constant and T is the matrix transpose operation;

[0035] (4.2.2) The feature fusion output Z after Transformer processing is obtained by processing through a feedforward network layer FFN composed of a multilayer perceptron. out .

[0036] More preferably, step (5) specifically includes the following steps:

[0037] (5.1) The fused comprehensive feature Z obtained from the second stage training output out Perform global average pooling g(·), then use a fully connected network (FC) to map the dimensions to the number of classes (2), resulting in a vector. As shown in the following formula:

[0038] Z = FC(g(Z) out ))

[0039] (5.2) Calculate the Softmax based on Z to obtain the final predicted score y′, and calculate the cross-entropy loss L for the two classes based on the label y. ceAs shown in the following formula:

[0040] L ce =y log y′+(1-y)log(1-y′)

[0041] (5.3) Construct the overall loss function L all As shown in the following formula:

[0042] L all =L ce +λL mask

[0043] Here, λ is a hyperparameter used to balance the cross-entropy loss and the mask loss.

[0044] The system for identifying deepfake faces based on rPPG multi-scale spatiotemporal graphs and a two-stage model, utilizing the aforementioned method, is characterized by the following features:

[0045] The rPPG multi-scale spatiotemporal graph generation module is used to calculate the rPPG spatiotemporal graph from face video frames;

[0046] The mask-guided local attention module is connected to the rPPG multi-scale spatiotemporal graph generation module to enhance the learning of local information and extract features of a single rPPG spatiotemporal graph.

[0047] The Transformer module, connected to the mask-guided local attention module, is used to fuse the combined features of multiple adjacent rPPG spatiotemporal graphs; and

[0048] The classification head module, connected to the Transformer module, is used to pool the fused integrated features and perform classification and recognition processing to obtain the identification result of the target image and construct the overall loss function.

[0049] The device for realizing deepfake face identification based on rPPG multi-scale spatiotemporal graphs and a two-stage model is characterized in that the device comprises:

[0050] A processor is configured to execute computer-executable instructions;

[0051] The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the various steps of the method for deepfake face identification based on rPPG multi-scale spatiotemporal graphs and a two-stage model described above.

[0052] The processor used to implement deepfake face identification based on rPPG multi-scale spatiotemporal graph and two-stage model is characterized in that the processor is configured to execute computer-executable instructions, which, when executed by the processor, implement the various steps of the method for deepfake face identification based on rPPG multi-scale spatiotemporal graph and two-stage model described above.

[0053] The computer-readable storage medium is characterized in that it stores a computer program that can be executed by a processor to implement the various steps of the method for deepfake face identification based on rPPG multi-scale spatiotemporal graph and two-stage model described above.

[0054] This invention employs a method, system, device, processor, and computer-readable storage medium for deepfake face identification based on multi-scale spatiotemporal maps of heart rate signals (rPPG) and a two-stage model. It innovatively uses multi-scale spatiotemporal maps of heart rate signals (rPPG) as model input and employs classic CNN models (e.g., EfficientNet) and Transformer as the two-stage model. To enhance the model's perception of local location information, this invention also innovatively introduces a mask-guided local attention module. Through pixel-level spatiotemporal map mask labels, the model is guided to further distinguish between different patterns of real and fake spatiotemporal maps. The Transformer module fuses features from multiple adjacent rPPG spatiotemporal maps through a self-attention mechanism. This technical solution was experimentally validated on the FaceForensics++ dataset, demonstrating significantly superior classification and recognition performance compared to baseline models. Attached Figure Description

[0055] Figure 1 This is a schematic diagram illustrating the generation process of the method for deepfake face identification based on rPPG multi-scale spatiotemporal graph and two-stage model according to the present invention.

[0056] Figure 2 This is a schematic diagram illustrating the process of generating multi-scale rPPG spatiotemporal graphs for the method of deepfake face identification based on rPPG multi-scale spatiotemporal graphs and a two-stage model according to the present invention.

[0057] Figure 3 This is a schematic diagram of the framework structure of the system for deepfake face identification based on rPPG multi-scale spatiotemporal graph and two-stage model according to the present invention.

[0058] Figure 4 This is a schematic diagram of the Transformer module of the present invention. Detailed Implementation

[0059] To more clearly describe the technical content of the present invention, the following description is provided in conjunction with specific embodiments.

[0060] Before describing the embodiments of the present invention in detail, it should be noted that, in the following, the terms “comprising,” “including,” or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0061] Please see Figure 1 As shown, this method for identifying deepfake faces based on rPPG multi-scale spatiotemporal graphs and a two-stage model includes the following steps:

[0062] The method for identifying deepfake faces based on rPPG multi-scale spatiotemporal graphs and a two-stage model is characterized by the following steps:

[0063] (1) Collect a dataset of deepfake face videos and perform preprocessing operations on the video data to obtain a set of cropped face video frames;

[0064] (2) Generate an rPPG multi-scale spatiotemporal map based on the cropped face video frames;

[0065] (3) Construct a mask-guided local attention module, perform the first stage of training, and extract features of a single rPPG spatiotemporal graph;

[0066] (4) Construct the Transformer module, conduct the second stage training, and integrate the comprehensive features of multiple adjacent rPPG spatiotemporal graphs;

[0067] (5) Construct a classification head, pool the fused high-dimensional features and perform classification and recognition processing to obtain the identification results of the target image and construct the overall loss function.

[0068] In practical applications, step (1) above is specifically as follows:

[0069] Download the FaceForensic++ dataset from the official dataset website to obtain the original video, extract images from it, and use a face extractor to obtain cropped face images;

[0070] In practical applications, as a preferred embodiment of the present invention, step (2) specifically includes the following steps:

[0071] (2.1) Divide a complete video into several 64-frame video segments with a step size of 16;

[0072] (2.2) For each frame, perform face alignment and extract facial key points;

[0073] (2.3) Based on facial key points, select 6 heartbeat signal information regions to form a set of ROIs R. t ={R 1t ,R 2t ,…,R nt};

[0074] (2.4) For the aforementioned ROI set R t For all non-empty subsets, calculate the mean of all pixels contained in each non-empty subset to obtain 2. 6 -1 represents the average value of 63 pixels across the RGB three channels;

[0075] (2.5) For each video segment, the same operation as (2.2)-(2.4) is performed on its 64 frames to obtain a multi-scale spatiotemporal graph with dimensions of 64×63×3, where 64 is the time length, 63 is the number of combinations of different information regions, and 3 is the number of RGB channels.

[0076] In a preferred embodiment of the present invention, step (3) includes the following steps:

[0077] (3.1) Construct EfficientNet as the backbone network f(·) for the input rPPG spatiotemporal graph X∈R 3×64×63 Features are extracted through the backbone network, resulting in the mid-layer feature map Fm = f. mid (X)∈R C×H×W Where C, H, and W represent the number of channels, columns, and rows of the feature map, respectively;

[0078] (3.2) Construct a mask-guided local attention module, using the mid-layer feature map F m As input, we obtain the attention mask A. mask :

[0079] A mask =Sigmoid(Conv(F m ))

[0080] Where Conv(·) represents the convolution operation;

[0081] (3.3) Integrate the attention mask with the mid-layer feature map F m Dot product yields the position-weighted feature map F′=A mask ·F m And use F′ as input for feature extraction in subsequent network layers;

[0082] (3.4) Calculate the pixel-level mask label A of the aforementioned rPPG spatiotemporal graph. gtFor rPPG spatiotemporal graphs generated from fake videos, find their corresponding real rPPG spatiotemporal graphs, perform pixel-by-pixel subtraction to obtain residual spatiotemporal graphs, then convert the residual spatiotemporal graphs to grayscale, normalize them to 0 to 1, and adjust their size to match the attention mask A. mask For masks of the same size, binarization with a threshold of 0.1 yields the corresponding pixel-level mask label A. gt ;

[0083] (3.5) Apply the attention mask A mask and the corresponding pixel-level mask label A gt Calculate the L1 distance using the following formula, and use it as the mask loss function L. mask :

[0084]

[0085] In a preferred embodiment of the present invention, step (4) specifically includes the following steps:

[0086] (4.1) Constructing the input sequence Z of the Transformer in :

[0087] Inputting K temporally adjacent rPPG spatiotemporal maps into the backbone network trained in the first stage yields K global high-dimensional features F. h Then, global average pooling is performed, and classification encoding and one-dimensional learnable positional encoding are superimposed as the input sequence Z of the Transformer. in ;

[0088] (4.2): Construct a two-stage Transformer model to obtain the comprehensive features of multiple adjacent rPPG spatiotemporal graphs:

[0089] Input sequence Z in A query matrix is ​​generated through a linear mapping layer. A key matrix and a Value matrix Next, the three matrices are passed into the multi-head self-attention mechanism (MSA) as shown in the following formula:

[0090]

[0091] Where T is the matrix transpose operation and d is the normalization constant. The feature fusion output Z is then processed by a feedforward network layer (FFN) composed of a multilayer perceptron to obtain the Transformer-processed feature fusion output. out .

[0092] In a preferred embodiment of the present invention, step (5) specifically comprises:

[0093] The fused features are then subjected to global average pooling, and a fully connected network (FC) is used to map the number of dimensions to the number of classes (2), resulting in... Calculate the final predicted score y′ based on Z, and calculate the binary classification cross-entropy loss L based on the label y. ce Finally, the overall loss function L is constructed. all As shown in the following formula:

[0094] Z = FC(g(Z) out ))

[0095] L ce =y log y′+(1-y)log(1-y′)

[0096] L all =L ce +λL mask

[0097] Here, λ is a hyperparameter used to balance the cross-entropy loss and the mask loss.

[0098] Please see Figure 3 As shown, this system for identifying deepfake faces based on rPPG multi-scale spatiotemporal graphs and a two-stage model utilizes the aforementioned method. The system includes:

[0099] The rPPG multi-scale spatiotemporal graph generation module is used to calculate the rPPG spatiotemporal graph from face video frames;

[0100] The mask-guided local attention module is connected to the rPPG multi-scale spatiotemporal graph generation module to enhance the learning of local information and extract features of a single rPPG spatiotemporal graph.

[0101] The Transformer module, connected to the mask-guided local attention module, is used to fuse the combined features of multiple adjacent rPPG spatiotemporal graphs; and

[0102] The classification head module, connected to the Transformer module, is used to pool the fused integrated features and perform classification and recognition processing to obtain the identification result of the target image and construct the overall loss function.

[0103] In a specific embodiment of the present invention, the classification and recognition method using this technical solution was tested as follows:

[0104] (1) Experimental Dataset

[0105] This invention uses the deep face forgery tool FaceForensics++ (FF++) for experimental verification. The FF++ dataset includes 1000 original videos, of which 720 were used for training and 280 for testing and validation. Each video was forged using four different face manipulation methods: DeepFakes (DF), Face2Face (F2F), FaceSwap (FS), and NeuralTextures (NT). Two of these methods replace the entire face (DF and FS), while the other two methods only process local areas around the mouth or eyes (F2F and NT).

[0106] (2) Training process

[0107] The initial learning rate was set to 1e-2, the SGD optimizer was used for learning, the batch size was set to 32, and the training lasted for 30 epochs.

[0108] (3) Test Results

[0109] In this embodiment, the method is first trained and tested on four subsets of FF++ datasets to evaluate its ability to classify binary classes (true and false). Then, multi-class classification training and testing are performed on these datasets with five classes, and Accuracy (Acc.) is selected as the algorithm's evaluation metric. The experimental results are shown in Table 1.

[0110] Table 1. Model performance (%) on the FF++ dataset

[0111]

[0112] As shown in Table 1, this embodiment demonstrates excellent performance in both true / false binary classification and multi-class classification with five categories when using the FF++ dataset as training samples, showcasing the effectiveness of the algorithm.

[0113] This device for identifying deepfake faces based on rPPG multi-scale spatiotemporal graphs and a two-stage model includes:

[0114] A processor is configured to execute computer-executable instructions;

[0115] The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the various steps of the method for deepfake face identification based on rPPG multi-scale spatiotemporal graphs and a two-stage model described above.

[0116] The processor for deepfake face identification based on rPPG multi-scale spatiotemporal graph and two-stage model is configured to execute computer-executable instructions. When executed by the processor, the computer-executable instructions implement the various steps of the method for deepfake face identification based on rPPG multi-scale spatiotemporal graph and two-stage model.

[0117] The computer-readable storage medium contains a computer program that can be executed by a processor to implement the various steps of the method for deepfake face identification based on rPPG multi-scale spatiotemporal graphs and a two-stage model described above.

[0118] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0119] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution device.

[0120] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0121] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0122] In the description of this specification, references to terms such as "an embodiment," "some embodiments," "example," "specific example," or "embodiment," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0123] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

[0124] This invention employs a method, system, device, processor, and computer-readable storage medium for deepfake face identification based on multi-scale spatiotemporal maps of heart rate signals (rPPG) and a two-stage model. It innovatively uses multi-scale spatiotemporal maps of heart rate signals (rPPG) as model input and employs classic CNN models (e.g., EfficientNet) and Transformer as the two-stage model. To enhance the model's perception of local location information, this invention also innovatively introduces a mask-guided local attention module. Through pixel-level spatiotemporal map mask labels, the model is guided to further distinguish between different patterns of real and fake spatiotemporal maps. The Transformer module fuses features from multiple adjacent rPPG spatiotemporal maps through a self-attention mechanism. This technical solution was experimentally validated on the FaceForensics++ dataset, demonstrating significantly superior classification and recognition performance compared to baseline models.

[0125] In this specification, the invention has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and variations can be made without departing from the spirit and scope of the invention. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

Claims

1. A method for identifying deepfake faces based on rPPG multi-scale spatiotemporal graphs and a two-stage model, characterized in that, The method includes the following steps: (1) Collect a dataset of deepfake face videos and perform preprocessing operations on the video data to obtain a set of cropped face video frames; (2) Generate an rPPG multi-scale spatiotemporal map based on the cropped face video frames; (3) Construct a mask-guided local attention module, perform the first stage of training, and extract features of a single rPPG spatiotemporal graph; (4) Construct the Transformer module, perform the second stage training, and integrate the comprehensive features of multiple adjacent rPPG spatiotemporal graphs; (5) Construct a classification head, pool the fused high-dimensional features and perform classification and recognition processing to obtain the identification results of the target image and construct the overall loss function; Step (3) specifically includes the following steps: (3.1) Construct EfficientNet as the backbone network For the input rPPG spatiotemporal graph Features are extracted through the backbone network, resulting in mid-layer feature maps. Where C, H, and W represent the number of channels, columns, and rows of the feature map, respectively; (3.2) Construct a mask-guided local attention module, using mid-layer feature maps Given the input, obtain the attention mask. : in, This represents the convolution operation; (3.3) Combine the attention mask with the mid-layer feature map Perform dot product processing to obtain the position-weighted feature map. and with The input is used for feature extraction in subsequent network layers; (3.4) Calculate the pixel-level mask labels of the aforementioned rPPG spatiotemporal graph. For rPPG spatiotemporal graphs generated from fake videos, find the corresponding real rPPG spatiotemporal graph, perform pixel-by-pixel subtraction to obtain the residual spatiotemporal graph, then convert the residual spatiotemporal graph to grayscale, normalize it to 0 to 1, and adjust its size to match the attention mask. For masks of the same size, binarization with a threshold of 0.1 yields the corresponding pixel-level mask labels. ; (3.5) Apply the attention mask mentioned above and corresponding pixel-level mask labels Calculate the L1 distance using the following formula, and use it as the mask loss function. : 。 2. The method for identifying deepfake faces based on rPPG multi-scale spatiotemporal graphs and a two-stage model according to claim 1, characterized in that, Step (2) includes the following steps: (2.1) Divide a complete video into steps. Frames are divided into several A video clip of a frame; (2.2) For each frame, perform face alignment and extract facial key points; (2.3) Based on the aforementioned facial key points, select n heartbeat signal information regions to form a set of ROIs. ; (2.4) For the aforementioned ROI set Given all non-empty subsets, calculate the mean of all pixels contained in each non-empty subset, and obtain... The average pixel value of each RGB three-channel; (2.5) For each video segment, it contains Each frame undergoes steps (2.2) to (2.4) to obtain a dimension of Multiscale spatiotemporal diagrams, in which For the length of time, 3 represents the number of different information area combinations, and 3 represents the number of RGB channels.

3. The method for identifying deepfake faces based on rPPG multi-scale spatiotemporal graphs and a two-stage model according to claim 1, characterized in that, Step (4) specifically includes the following steps: (4.1) Input the K temporally adjacent rPPG spatiotemporal maps into the backbone network trained in the first stage to obtain K global high-dimensional features. Then, global average pooling is performed, and classification encoding and one-dimensional learnable positional encoding are superimposed as the input sequence for the Transformer. ; (4.2) Construct a feature fusion module for multiple rPPG spatiotemporal graphs based on Transformer: The input sequence... Multi-head self-attention (MSA) operations are performed, followed by a feedforward network (FFN). After each operation, layer normalization (LN) and residual connections are used to further adjust the output, resulting in the Transformer output. .

4. The method for identifying deepfake faces based on rPPG multi-scale spatiotemporal graphs and a two-stage model according to claim 1, characterized in that, Step (4.2) specifically includes the following steps: (4.2.1) Input sequence A query matrix is ​​generated through a linear mapping layer. A key matrix and a Value matrix Then, the three matrices are passed into the multi-head self-attention mechanism MSA, as shown in the following formula: Where d is the normalization constant and T is the matrix transpose operation; (4.2.2) The feature fusion output after Transformer processing is obtained by processing through a feedforward network layer (FFN) composed of a multilayer perceptron. .

5. The method for identifying deepfake faces based on rPPG multi-scale spatiotemporal graphs and a two-stage model according to claim 1, characterized in that, Step (5) specifically includes the following steps: (5.1) The fused comprehensive features obtained from the second stage training output Perform global average pooling Then, a fully connected network (FC) is used to map the dimension to the number of categories (2), resulting in a vector. As shown in the following formula: (5.2) For vectors The final prediction score is obtained by calculating Softmax. And according to the label Calculate the cross-entropy loss for two classes. As shown in the following formula: (5.3) Construct the overall loss function As shown in the following formula: in, It is a hyperparameter used to balance cross-entropy loss and mask loss.

6. A system for identifying deepfake faces based on rPPG multi-scale spatiotemporal graphs and a two-stage model using the method described in any one of claims 1 to 5, characterized in that, The system includes: The rPPG multi-scale spatiotemporal graph generation module is used to calculate the rPPG spatiotemporal graph from face video frames; The mask-guided local attention module is connected to the rPPG multi-scale spatiotemporal graph generation module to enhance the learning of local information and extract features of a single rPPG spatiotemporal graph. The Transformer module, connected to the mask-guided local attention module, is used to fuse the comprehensive features of multiple adjacent rPPG spatiotemporal graphs; and The classification head module, connected to the Transformer module, is used to pool the fused integrated features and perform classification and recognition processing to obtain the identification result of the target image and construct the overall loss function.

7. An apparatus for identifying deepfake faces based on rPPG multi-scale spatiotemporal graphs and a two-stage model, characterized in that, The device includes: A processor is configured to execute computer-executable instructions; The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the method for deepfake face identification based on rPPG multi-scale spatiotemporal graphs and a two-stage model as described in any one of claims 1 to 5.

8. A processor for implementing deepfake face identification based on rPPG multi-scale spatiotemporal graphs and a two-stage model, characterized in that, The processor is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the method for deepfake face identification based on rPPG multi-scale spatiotemporal graph and two-stage model as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the various steps of the method for deepfake face identification based on rPPG multi-scale spatiotemporal graph and two-stage model as described in any one of claims 1 to 5.

Images