Face-spoofing video detection method based on multi-temporal attention network

Abstract

The application discloses a face fake video detection method based on a multi-time sequence attention network, wherein an attention feature map from a frame in an inspection sequence is taken as feature map weight information through an inspection branch network, and a convolution operation is used to detect artifacts in a frame to be processed in the inspection sequence; a browsing branch network is used to detect the continuity of a frame to be processed in a browsing sequence through a convolution operation, and the output of the inspection branch network at each layer is fused through lateral connection; finally, the output results of the convolution processing of the browsing branch network and the inspection branch network are fused through a full connection layer, and finally, an identification result is output, so that the effectiveness of face detection is ensured and the detection efficiency is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of fake video detection, in particular to a face fake video detection method based on a multi-temporal attention network. BACKGROUND

[0002] The face fake video detection technology is that an input device (a local camera, a Web camera, etc.) inputs data (including video streaming media, image sequences, images, videos, etc.) into a trained face detector, performs frame-by-frame face detection on the input data by using the trained face detector, extracts a face region in each frame of image, and inputs the extracted face region into a binary classification convolutional network (Real / Fake) to identify the authenticity of the input face region by a face forgery discrimination model.

[0003] With the continuous development of GAN generation network technology, the generated face fake images are increasingly realistic, and the deep fake detection technology for video data has attracted more and more attention in recent years. In the fake detection and security prevention system, the face in the video stream needs to be detected to identify the authenticity of the face to meet the user's requirement for face fake video identification.

[0004] However, the current detection method either only focuses on the information inside the video frame or is too sensitive to the disturbance between video frames. In short, the conventional face fake video detection technology described above has been difficult to adapt to the detection requirement, and the existing face fake detection technology still needs to be further improved to further meet the detection requirement. SUMMARY

[0005] The present application aims at the technical defects in the prior art and provides a face fake video detection method based on a multi-temporal attention network.

[0006] To achieve the purpose of the present application, the technical solution adopted is:

[0007] A face fake video detection method based on a multi-temporal attention network, comprising the following steps:

[0008] Performing face detection and tracking on the input video sequence from the starting frame, and storing the same face as a face sequence to be processed after recognition;

[0009] The network down-samples the face sequence at different temporal sampling rates to obtain a browsing sequence and a scrutiny sequence;

[0010] Extracting an attention feature map from the frame in the scrutiny sequence;

[0011] After the feature map of each frame in the browsing sequence is extracted by the browsing branch network through the convolution operation each time, the feature map is fused with the feature map output by each layer of the viewing branch network through the lateral connection, and the fused feature map is taken as the input of the next convolution of the browsing branch network; after the feature map of each frame in the viewing sequence is extracted by the viewing branch network through the convolution operation each time, the feature map is multiplied with the corresponding attention feature map and then output;

[0012] The full connection layer fuses the output results of the convolution processing of the browsing branch network and the viewing branch network respectively, and finally outputs the recognition result.

[0013] Preferably, the convolution kernel size of each layer of the browsing branch network and the viewing branch network subjected to the convolution operation is the same.

[0014] Preferably, the number of convolution kernels of each layer of the browsing branch network subjected to the convolution operation is β times the number of convolution kernels of each layer of the viewing branch network, and α×β=1, the length of the browsing sequence is α times the length of the viewing sequence.

[0015] Preferably, the lateral connection fusion adopts dense convolution fusion, and the implementation steps are as follows:

[0016] The feature map formed by the browsing branch network is processed by a 3D convolution network to reduce the number of feature maps of each frame, and then the obtained feature map is connected with the feature map formed by the viewing branch network according to the corresponding relationship, fused to obtain the corresponding fused feature map.

[0017] Preferably, the sampling interval of the 3D convolution network processing the feature map formed by the browsing branch network is 1 / α, and one frame is taken every 1 / α frames for processing.

[0018] Preferably, a pre-trained network is used to extract the attention feature map of the frame in the viewing sequence.

[0019] More preferably, a pre-trained U-Net network is used to extract the attention feature map of the frame in the viewing sequence.

[0020] Preferably, when the U-Net network is trained, based on the corresponding relationship between the real face and the fake face in the data set, the difference between the positive and negative samples is obtained to obtain the fake area of the fake face to the real face, so as to obtain the attention feature map of the frame.

[0021] Preferably, when the U-Net network is trained, a soft attention label is introduced, and the soft attention label is taken as the label of the real face image to replace the real face image for network training. The real face image is represented as R Image , the compression rate of the real face image is ρ (0<ρ<1), and the soft label of the real face image is represented as S label , ρ×R ImageThe expression of the soft label of the real face image is as follows when the real face image is compressed ρ times, that is, the resolution of the real face image is compressed from (W, H) to (ρ×W, ρ×H) :

[0022] S label = (ρ×R Image )×(1 / ρ)-R Image .

[0023] Preferably, in the detection process, the scanning frame rate of the browsing branch network on the face sequence is greater than the scanning frame rate of the reviewing branch network.

[0024] The fake video detection technology of the application detects the artifacts in the frames to be processed in the review sequence through a review branch network by taking the attention feature map from the frames in the review sequence as feature map weight information and through convolution operation; the browsing branch network detects the coherence of the frames to be processed in the browsing sequence through convolution operation, and the output of each layer of the review branch network is fused through lateral connection; finally, the output results of the convolution processing of the browsing branch network and the review branch network are fused by the full connection layer, and the final recognition result is output, which ensures the effectiveness of face detection and greatly improves the detection efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is the flowchart of the face fake video detection method based on the multi-time sequence attention network of the application.

[0026] Figure 2 is the structural schematic diagram of the multi-time sequence fusion network of the application.

[0027] Figure 3 is the pre-training schematic diagram of the U-Net network structure used in the embodiment of the application.

[0028] Figure 4 is the schematic diagram of the fusion of the browsing branch and the review branch using different fusion modes.

[0029] Figure 5 is the schematic diagram of the result obtained by subtracting the real image from the image after the resolution of the image is reduced and then expanded to the original resolution under different soft attention label settings in the network training process.

[0030] Figure 3 The parameters are as follows:

[0031] 3x3 Conv, F filters*, 3x3 Conv represents 3x3 convolution, and * represents that the convolution operation is followed by BatchNormalization regularization and ReLU operation;

[0032] 2x2 Maxpooling2D means using a 2x2 convolutional kernel for max pooling; Concatenation means concatenating the channels of two convolutional kernels.

[0033] 3x3 Transpose Conv means performing deconvolution with a 3x3 convolution kernel;

[0034] 2x2 UpSamling2D means using a 2x2 convolution kernel to perform upsampling on the feature map to perform super-resolution. Detailed Implementation

[0035] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0036] like Figure 1 As shown in the figure, the detection process of the face spoofing video detection method based on multi-temporal attention network in this invention is as follows:

[0037] The input video sequence is subjected to face detection and tracking starting from the first frame. Once the same face is identified, it is stored as a face sequence to be processed.

[0038] The network downsamples the face sequence at different temporal sampling rates to obtain browsing and review sequences.

[0039] Extract attention feature maps from frames in the examined sequence;

[0040] After each convolution operation, the browsing branch network extracts the feature map of the frame in the browsing sequence and fuses it with the feature map output from each layer of the review branch network through a lateral connection. The fused feature map is used as the input for the next convolution of the browsing branch network. After each convolution operation, the review branch network extracts the feature map of the frame in the review sequence and multiplies it with the corresponding attention feature map before outputting it.

[0041] The fully connected layer fuses the outputs of the convolutional processing of the browsing branch network and the examining branch network, and finally outputs the recognition result.

[0042] In embodiments of the present invention, the input video sequence may be video data, streaming media data, image sequences, camera capture sequences, or other data.

[0043] In an embodiment of the present invention, after receiving the input video sequence, the network further determines whether the format of the input sequence is correct. If it is correct, it performs decoding; if it is incorrect, it returns an error code until it receives input data with the correct format, and then performs decoding processing.

[0044] Further, in the embodiment of the present application, after detecting different faces, the different faces are stored as different sequences according to the recognition, forming face sequences of different ID sequences, and then the face sequences are subjected to authenticity detection processing.

[0045] Among them, the input sequence can be subjected to face detection from the starting frame, then the detected face is tracked, the key points of the face are detected, the face is recognized, and the different face sequences recognized are stored according to the recognized ID.

[0046] After the face sequence is completely detected, the detection technology of the embodiment of the present application can display the authenticity result in a visual manner, such as color marking or text marking, and other display methods that can realize visual display, such as output display by a display device, for example, a display screen.

[0047] Among them, when the attention feature map of the frame in the review sequence is extracted, that is, the attention information of the forged area of the frame is extracted, a pre-trained extraction network can be used, such as U-Net, but it is not limited to the use of this network, and other segmentation networks such as mask-rcnn can also be used.

[0048] The attention feature map is used as the weight information of the feature map of the review branch network, and the review branch network uses it to process the feature map and detect the artifacts in the feature map.

[0049] After obtaining the attention feature map, the resolution of the feature map is adjusted, and the attention feature map is added as weight information to the review branch network, so that the network can give higher weight to the face or face forgery area, thereby improving the identification ability of the network model.

[0050] It should be noted that the U-Net network described above is pre-trained, and since there is a corresponding relationship between the real face and the forged face in the data set, the difference between the positive and negative samples can be obtained. The tampered area of the forged face to the real face is the Attention area. Adding the tampered area as the Attention area to the network can increase the attention ability of the network to the tampered area.

[0051] Reference Figure 5As shown, the series of images are false artifacts generated after the images are scaled by different compression rates, and similar to the artifacts of the fake images, in which real represents the real image, fake represents the fake image, diff represents the result of the real image minus the fake image in the gray channel, black is a pure black image, p represents the scaling rate of the image, p-0.2 represents the image after being scaled to 0.2 times the original resolution, and then subtracting the real image.

[0052] As described above, since the false artifacts generated after the images are scaled by different compression rates are similar to the artifacts of the fake images, in order to avoid the network from making false judgments due to the false artifacts caused by image scaling and compression, and to make the network pay more attention to the identification of the fake video, a soft attention label (Soft Attention Lable) is introduced, and the network is trained through the application of the soft attention label to improve the identification ability of the network.

[0053] For example, during training, the soft attention label (Soft Attention Lable) can be used instead of a pure black image as a label of a real face, and the real face is represented as R Image , the image compression rate is p (0 < p < 1), the soft label of the real image is S label , and p x R Image represents that the image is compressed by p times, i.e., the resolution of the image is compressed from (W, H) to (p x W, p x H).

[0054] The expression of the soft label is as follows:

[0055] S label = (p x R Image ) x (1 / p) - R Image .

[0056] During the pre-training process of the network, the decoded fake video can be represented as {V f1 , V f2 , V f3 , ···, V fn}, the decoded real image corresponding to the fake video can be represented as {V r1 , V r2 , V r3 , ···, V rn}, and for the fake sequence, the feature map of the input frame can be represented as {V r1 -V f1 , V r2 -V f2 , V r3 -V f3 , ···, Vrn -V fn}, use {V att1 ,V att2 ,V att3 ,···,V attn} to represent.

[0057] In this way, different fake videos correspond to the attention feature map of the real image, which can be used as attention weight in the detection of the review branch network in the future.

[0058] In the embodiment of the application, the review branch network and the browsing branch network can be implemented by using a convolution network, and can include multiple layers of convolution operations.

[0059] In the embodiment of the application, in the detection process, the browsing branch network scans the entire video at a high frame rate, and pays attention to the continuity of the video in the browsing sequence in time sequence; the review branch network analyzes the key frames in the video at a low frame rate, and pays attention to whether there is an artifact in the frame in the review sequence.

[0060] Through such a technology, the continuity detection and artifact detection in the network processing are performed in different channels, and the continuity sequence and the artifact detection can be corresponded, so that a fast and accurate detection result can be obtained subsequently.

[0061] In the embodiment of the application, as shown in Figure 2 After the video decoding sequence of the input network is serialized, the browsing sequence and the review sequence can be obtained through different time sequence sampling rates, and Temporal Stride represents time sequence sampling of the input sequence. The browsing branch network processes more frame sequences with fewer network parameters. The review branch network processes fewer frame sequences with more network parameters.

[0062] Meanwhile, a pre-trained U-Net network extracts the attention feature map (Attention map) of the frame in the review sequence, sends the extracted attention feature map (Attention map) into the review branch network, and uses the attention feature map (Attention map) as attention weight to process the frame in the review sequence by the review branch network, extracts the feature map, and the addition of the attention feature map (Attention map) can make the review branch network better distinguish the authenticity of the video frame. After each layer of convolution, the output results of the review branch network and the browsing branch network are fused through lateral connection (lateral connection) at the corresponding nodes, and the final result is used as the basis for discrimination. The feature map finally output by the review branch network and the browsing branch network is fused and processed through a fully connected layer (Fully Connect Layer), and the detection result (Result) is output.

[0063] The U-Net network extracts an attention feature map from the review sequence, obtains an attention sequence, then sends the attention weight into the review branch network, then performs scale reduction on the attention sequence according to the convolution step corresponding to the review sequence, and sends the attention weight formed after the scale reduction to the review branch network, so that the corresponding convolution layer of the review branch network performs convolution processing on the feature map in the convolution layer based on the attention weight.

[0064] The U-Net network is used to extract the attention feature map by adopting convolution, maximum pooling and Concatenation connection and other processing means on the input face sequence, and can refer to Figure 3 .

[0065] Referring to Figure 2 , during processing, the input video and streaming media data are decoded to obtain T x τ b independent frames, the review branch network extracts one frame every τ b frames to obtain a review sequence with a length of T. If the length of the review sequence is α times the length of the review sequence, then the review branch network extracts one frame every τ s frames (τ s = α x τ b ) to obtain a review sequence with a length of T / α.

[0066] In order to maintain that the review branch network and the review branch network can be fused through side connection, in the embodiment of the application, the convolution kernel size of the review branch network and the review branch network is maintained to be the same, the number of convolution kernels of each layer of the review branch network is β times that of the review branch network (α x β = 1), and the number of feature map outputs of the review branch network is β times that of the review branch.

[0067] According to the above, the output of the review branch network can be represented as {T, W, H, C}, and the output of the review branch network can be represented as {αT, W, H, βC}, wherein W and H represent the width and height of the feature map, and because α x β = 1, the review branch network and the review branch network can be fused through side connection.

[0068] In the embodiment of the application, the output feature map of the review branch network and the output feature map of the review branch network can be fused by using any fusion mode of channel reorganization fusion, channel down-sampling fusion, dense convolution fusion or sparse convolution fusion.

[0069] The channel reorganization fusion mode is:

[0070] If the shape of the feature map in the browsing branch network can be represented as {T, W, H, C}, the feature map channels in the browsing branch network are decomposed, the number of feature maps and the number of feature map channels are recombined, and the number of feature map channels in the browsing branch network is adjusted to be consistent with the number of feature map channels in the review branch network {αT, W, H, βC}, then the feature maps in the two branch networks are added one by one to obtain the final channel recombination and fusion feature map.

[0071] The channel downsampling fusion manner is:

[0072] If the shape of the feature map in the browsing branch network can be represented as {T, W, H, C}, the feature map channels in the browsing branch network are decomposed, the number of feature maps and the number of feature map channels are recombined, and the number of feature map channels in the browsing branch network is adjusted to be consistent with the number of feature map channels in the review branch network {αT, W, H, βC}, then the feature maps in the two branch networks are added one by one to obtain the final channel recombination and fusion feature map.

[0073] The dense convolution fusion manner is:

[0074] If the shape of the feature map in the browsing branch network can be represented as {T, W, H, C}, the feature map channels in the browsing branch network are decomposed, the number of feature maps and the number of feature map channels are recombined, and the number of feature map channels in the browsing branch network is adjusted to be consistent with the number of feature map channels in the review branch network {αT, W, H, βC}, then the feature maps in the two branch networks are added one by one to obtain the final channel recombination and fusion feature map.

[0075] The sparse convolution fusion manner is:

[0076] If the shape of the feature map in the browsing branch network can be represented as {T, W, H, C}, the feature map channels in the browsing branch network are decomposed, the number of feature maps and the number of feature map channels are recombined, and the number of feature map channels in the browsing branch network is adjusted to be consistent with the number of feature map channels in the review branch network {αT, W, H, βC}, then the feature maps in the two branch networks are added one by one to obtain the final channel recombination and fusion feature map.

[0077] To verify the performance of the browsing branch network and the review branch network in different side connection manners, the performance of four kinds of side connection manners is tested under the same experimental conditions, in the experiment, α=0.25, β=4, k=4, and after verification, the best side connection manner is dense convolution fusion.

[0078] Compared with the mode without side connection, the dense convolution fusion mode improves the performance by 8.83% in the experiment, and even the mode of using channel fusion for side connection has a performance improvement of 6.65% compared with the network without side connection.

[0079]

[0080] Table 1

[0081] The forged video detection technology of the present application takes the attention feature map from the frame of the review sequence as feature map weight information through a review branch network, detects the artifacts in the frame to be processed in the review sequence through convolution operation; through a browsing branch network, the coherence of the frame to be processed in the browsing sequence is detected through convolution operation, and the output of each layer of the review branch network is fused through side connection at the same time; finally, the output results of the convolution processing of the browsing branch network and the review branch network are fused by the full connection layer, and finally the recognition result is output, which guarantees the effectiveness of face detection and greatly improves the detection efficiency.

[0082] The above only describes the preferred embodiments of the present application, and it should be pointed out that for ordinary skilled persons in the art, some improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A method for detecting face-spoofing videos based on a multi-temporal attention network, characterized in that, The method comprises the following steps: Carrying out face detection and tracking on an input video sequence from a starting frame, and storing a same face as a to-be-processed face sequence after recognition; A network reduces the sampling rate of the face sequence at different times to obtain a browsing sequence and a scrutiny sequence; Extracting an attention feature map from a frame in the scrutiny sequence; After the browsing branch network extracts a feature map of a frame in the browsing sequence through convolution operation each time, the feature map is fused with a feature map output by each layer of the scrutiny branch network through lateral connection, and the fused feature map is used as an input of the next convolution of the browsing branch network; after the scrutiny branch network extracts a feature map of a frame in the scrutiny sequence through convolution operation each time, the feature map is multiplied with a corresponding attention feature map and then output; A fully connected layer fuses output results of convolution processing of the browsing branch network and the scrutiny branch network respectively, and finally outputs a recognition result. 2.The method of claim 1, wherein, The browsing branch network and the scrutiny branch network have the same kernel size for each layer of convolution operation. 3.The method of claim 2, wherein, The number of convolution kernels of each layer of the browsing branch network is β times the number of convolution kernels of each layer of the scrutiny branch network, and α×β=1, the length of the browsing sequence is α times the length of the scrutiny sequence.

4. The method of claim 1, wherein the method comprises: The lateral connection fusion adopts dense convolution fusion, and the implementation steps are as follows: The feature map formed by the browsing branch network is processed by a 3D convolution network to reduce the number of feature maps of each frame, and then the obtained feature map is connected with the feature map formed by the scrutiny branch network according to the corresponding relationship, fused to obtain the corresponding fusion feature map.

5. The method of claim 4, wherein the method comprises: The sampling interval of the 3D convolution network processing the feature map formed by the browsing branch network is 1 / α, and one frame is taken every 1 / α frames for processing.

6. The method of claim 1, wherein the method further comprises: A pre-trained network is used to extract an attention feature map from a frame in the scrutiny sequence.

7. The method of claim 6, wherein the method further comprises: A pre-trained U-Net network is used to extract an attention feature map from a frame in the scrutiny sequence. 8.The method of claim 7, wherein, During the training of the U-Net network, based on the corresponding relationship between the real face and the fake face in the data set, the difference between the positive and negative samples is obtained, the fake area of the fake face to the real face is obtained, and thus the attention feature map of the frame is obtained. 9.The method of claim 7 or 8, wherein, During training of the U-Net network, a soft attention label is introduced, and the soft attention label is used as a label of the real face image to replace the real face image for network training, so that the real face image is represented as R Image , the compression rate of the real face image is ρ (0 < ρ < 1), and the soft label of the real face image is represented as S label , and ρ × R Image represents that the real face image is compressed by ρ times, that is, the resolution of the real face image is compressed from (W, H) to (ρ × W, ρ × H), and the expression of the soft label of the real face image is as follows: S label = (p x R Image ) x (1 / p) - R Image . 10.The method of claim 1, wherein, During the detection process, the scanning frame rate of the browsing branch network on the face sequence is greater than the scanning frame rate of the scrutiny branch network.

Images