Method and system for detecting fake face videos based on multi-feature fusion

Abstract

The present invention relates to a method and system for detecting fake face videos based on multi-feature fusion, and to the field of target detection technology. The method comprises: performing frame decomposition on a face video to be detected to obtain a plurality of continuous face images to be detected; inputting the face images to be detected into a spatial feature extraction network to obtain a fused feature map; and inputting the plurality of fused feature maps into a temporal attention network to determine the type of face video. The spatial feature extraction network comprises a first fake feature branch, a second fake feature branch, a cross-modal interaction module and a feature fusion module; the cross-modal interaction module is used to extract image feature complementary information between the first fake feature branch and the second fake feature branch, and input the image feature complementary information into the first fake feature branch and the second fake feature branch accordingly. The present invention improves the accuracy of fake face video detection.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a method and system for detecting forged human face videos based on multi-feature fusion. Background Art

[0002] With the development of artificial intelligence technology represented by deep learning, deep fake face technology has made a breakthrough, making it possible to tamper with or generate highly realistic and difficult to identify fake faces. On the one hand, deep fake face technology has its positive value in education, cultural creation, entertainment and other fields. On the other hand, the abuse of deep fake face technology may bring new challenges to national defense security and people's livelihood security.

[0003] At present, the defense against forged face videos is mostly based on passive detection, which obtains information or extracts features from the face video itself to identify forged face videos. If the forged video detection task is converted into a binary classification task, it can be summarized into the following five categories according to the different forgery mechanisms: 1) Texture-level detection methods, which focus on detecting the difference in forged face noise. 2) Attribute-level detection methods, which perform detection in the face airspace and focus on the details of the face. 3) Object-level detection methods, which focus on extracting local-global multi-scale forged features. 4) Structural-level detection methods, which mainly extract cross-scale consistency features within the frame. 5) Video frame-level detection methods, which take advantage of the inconsistent features between forged video frames.

[0004] Although the existing fake face detection technology based on deep learning technology has better detection performance and application potential than the detection technology based on manual design, it still has the following shortcomings: 1) The preprocessing of high-frequency signals and content details of fake face video frames is too simple, and there is a lack of effective enhancement of high-frequency details in fake face video frames. Considering that the deep video forgery algorithm is more about synthesizing low-frequency signals in the image, it is difficult to take into account the simulation of video content details and the real video noise distribution based on the physical principles of camera imaging. 2) Most detection methods only use inconsistencies within or between frames, and do not combine these two important forgery features well. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for detecting forged face videos based on multi-feature fusion, so as to improve the accuracy of forged face video detection.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] The present invention provides a method for detecting fake face videos based on multi-feature fusion, comprising:

[0008] Decompose the video of the face to be detected into frames to obtain multiple continuous images of the face to be detected;

[0009] Inputting the face image to be detected into a spatial feature extraction network to obtain a fused feature map;

[0010] Inputting the plurality of fused feature maps into a temporal attention network to determine the type of face video; the type of face video includes a normal face video and a forged face video;

[0011] The spatial feature extraction network includes a first forged feature branch, a second forged feature branch, a cross-modal interaction module and a feature fusion module; the first forged feature branch is used to perform feature extraction on the face image to be detected to obtain a first feature map; the second forged feature branch is used to perform high-pass preprocessing and feature extraction on the face image to be detected in sequence to obtain a second feature map;

[0012] The cross-modal interaction module is arranged between the first forged feature branch and the second forged feature branch, and is used to extract image feature complementary information between the first forged feature branch and the second forged feature branch, and input the image feature complementary information to the first forged feature branch and the second forged feature branch accordingly;

[0013] The input end of the feature fusion module is connected to the output end of the first forged feature branch and the output end of the second forged feature branch respectively, and the feature fusion module is used to perform feature fusion on the first feature map and the second feature map to obtain a fused feature map.

[0014] Optionally, the first forged feature branch includes a first feature extraction module and a third feature extraction module connected in sequence; the second forged feature branch includes a high-pass preprocessing module, a second feature extraction module and a fourth feature extraction module connected in sequence;

[0015] The first feature extraction module is used to perform RGB domain spatial feature extraction on the face image to be detected to obtain a first sub-feature map; the high-pass preprocessing module is used to perform high-frequency noise extraction on the face image to be detected to obtain a first noise map; the second feature extraction module is used to perform feature extraction on the first noise map to obtain a second sub-feature map;

[0016] The output end of the first feature extraction module is also connected to the first input end of the cross-modal interaction module; the output end of the second feature extraction module is also connected to the second input end of the cross-modal interaction module;

[0017] The first output end of the cross-modal interaction module is connected to the input end of the third feature extraction module; the second output end of the cross-modal interaction module is connected to the input end of the fourth feature extraction module; the cross-modal interaction module is used to interact with the first sub-feature graph and the second sub-feature graph, extract the first complementary information and the second complementary information, and then input the first complementary information to the third feature extraction module, and send the second complementary information to the fourth feature extraction module; the first complementary information is the distinguishing feature information of the second sub-feature graph relative to the first sub-feature graph; the second complementary information is the distinguishing feature information of the first sub-feature graph relative to the second sub-feature graph;

[0018] The third feature extraction module is used to determine a first feature map according to the first sub-feature map and the first complementary information; the fourth feature extraction module is used to determine a second feature map according to the second sub-feature map and the second complementary information;

[0019] The output end of the third feature extraction module and the output end of the fourth feature extraction module are both connected to the input end of the feature fusion module.

[0020] Optionally, the cross-modal interaction module includes a first splicing submodule, a convolution submodule, a maximum pooling submodule, an average pooling submodule, a second splicing submodule and a convolution activation submodule;

[0021] The first input end of the first splicing submodule is connected to the output end of the first feature extraction module, and the second input end of the first splicing module is connected to the output end of the second feature extraction module;

[0022] The output end of the first splicing submodule is connected to the input end of the convolution submodule; the first output end of the convolution submodule is connected to the input end of the maximum pooling submodule, and the second output end of the convolution submodule is connected to the input end of the average pooling submodule;

[0023] The output end of the maximum pooling submodule is connected to the first input end of the second splicing submodule, the output end of the average pooling submodule is connected to the second input end of the second splicing submodule; the output end of the second splicing submodule is connected to the input end of the convolution activation submodule;

[0024] The first output end of the convolution activation submodule is connected to the input end of the third feature extraction module, and the second output end of the convolution activation submodule is connected to the input end of the fourth feature extraction module.

[0025] Optionally, the training process of the spatial feature extraction network specifically includes:

[0026] According to the training set and the loss function, the calibration deep convolutional network is trained to obtain a spatial feature extraction network; the training set includes an initial face image and label information corresponding to the initial face image; the label information is the facial features of the face in the initial face image; the calibration deep convolutional network includes a first calibration forged feature branch, a second calibration forged feature branch, a calibration cross-modal interaction module, a calibration feature fusion module and a calibration classification module;

[0027] The input end of the first calibration forged feature branch is used to input the initial face image; the output end of the first calibration forged feature branch is connected to the first input end of the calibration feature fusion module;

[0028] The input end of the second calibration forged feature branch is used to input the initial face image; the output end of the second calibration forged feature branch is connected to the second input end of the calibration feature fusion module;

[0029] The calibration cross-modal interaction module is arranged between the first calibration forged feature branch and the second calibration forged feature branch;

[0030] The output end of the calibration fusion module is connected to the input end of the calibration classification module; the output end of the calibration classification module is used to output the facial features of the initial face image.

[0031] Optionally, the cross-modal interaction module includes a first interaction submodule, a second interaction submodule and a third interaction submodule; the first forged feature branch also includes a first feature extraction submodule and a third feature extraction submodule; the second forged feature branch also includes a second feature extraction submodule and a fourth feature extraction submodule;

[0032] The first input end of the first interaction submodule is connected to the output end of the first feature extraction module, and the first output end of the first interaction submodule is connected to the input end of the first feature extraction submodule;

[0033] The second input end of the first interaction submodule is connected to the output end of the second feature extraction module; the second output end of the first interaction submodule is connected to the input end of the second feature extraction submodule;

[0034] The first input end of the second interaction submodule is connected to the output end of the first feature extraction submodule, and the first output end of the second interaction submodule is connected to the input end of the third feature extraction submodule;

[0035] The second input end of the second interaction submodule is connected to the output end of the second feature extraction submodule; the second output end of the second interaction submodule is connected to the input end of the fourth feature extraction submodule;

[0036] The first input end of the third interaction submodule is connected to the output end of the third feature extraction submodule, and the first output end of the third interaction submodule is connected to the input end of the third feature extraction module;

[0037] The second input end of the third interaction submodule is connected to the output end of the fourth feature extraction submodule; the second output end of the third interaction submodule is connected to the input end of the fourth feature extraction module.

[0038] Optionally, the temporal attention network includes an attention Transformer module and a classification module based on a multi-layer perceptron connected in sequence.

[0039] To achieve the above object, the present invention also provides the following technical solutions:

[0040] A forged face video detection system based on multi-feature fusion, comprising:

[0041] A video decomposition unit is used to decompose the video of the face to be detected into frames to obtain multiple continuous images of the face to be detected;

[0042] A feature fusion unit, used for inputting the face image to be detected into a spatial feature extraction network to obtain a fused feature map;

[0043] A video type determination unit, used for inputting the plurality of fused feature maps into a temporal attention network to determine the type of face video; the face video type includes a normal face video and a forged face video;

[0044] The spatial feature extraction network includes a first forged feature branch, a second forged feature branch, a cross-modal interaction module and a feature fusion module; the first forged feature branch is used to perform feature extraction on the face image to be detected to obtain a first feature map; the second forged feature branch is used to perform high-pass preprocessing and feature extraction on the face image to be detected in sequence to obtain a second feature map;

[0045] The cross-modal interaction module is arranged between the first forged feature branch and the second forged feature branch, and is used to extract image feature complementary information between the first forged feature branch and the second forged feature branch, and input the image feature complementary information to the first forged feature branch and the second forged feature branch accordingly;

[0046] The input end of the feature fusion module is connected to the output end of the first forged feature branch and the output end of the second forged feature branch respectively, and the feature fusion module is used to perform feature fusion on the first feature map and the second feature map to obtain a fused feature map.

[0047] Optionally, the first forged feature branch includes a first feature extraction module and a third feature extraction module connected in sequence; the second forged feature branch includes a high-pass preprocessing module, a second feature extraction module and a fourth feature extraction module connected in sequence;

[0048] The first feature extraction module is used to perform RGB domain spatial feature extraction on the face image to be detected to obtain a first sub-feature map; the high-pass preprocessing module is used to perform high-frequency noise extraction on the face image to be detected to obtain a first noise map; the second feature extraction module is used to perform feature extraction on the first noise map to obtain a second sub-feature map;

[0049] The output end of the first feature extraction module is also connected to the first input end of the cross-modal interaction module; the output end of the second feature extraction module is also connected to the second input end of the cross-modal interaction module;

[0050] The first output end of the cross-modal interaction module is connected to the input end of the third feature extraction module; the second output end of the cross-modal interaction module is connected to the input end of the fourth feature extraction module; the cross-modal interaction module is used to interact with the first sub-feature graph and the second sub-feature graph, extract the first complementary information and the second complementary information, and then input the first complementary information to the third feature extraction module, and send the second complementary information to the fourth feature extraction module; the first complementary information is the distinguishing feature information of the second sub-feature graph relative to the first sub-feature graph; the second complementary information is the distinguishing feature information of the first sub-feature graph relative to the second sub-feature graph;

[0051] The third feature extraction module is used to determine a first feature map according to the first sub-feature map and the first complementary information; the fourth feature extraction module is used to determine a second feature map according to the second sub-feature map and the second complementary information;

[0052] The output end of the third feature extraction module and the output end of the fourth feature extraction module are both connected to the input end of the feature fusion module.

[0053] Optionally, the cross-modal interaction module includes a first splicing submodule, a convolution submodule, a maximum pooling submodule, an average pooling submodule, a second splicing submodule and a convolution activation submodule;

[0054] The first input end of the first splicing submodule is connected to the output end of the first feature extraction module, and the second input end of the first splicing module is connected to the output end of the second feature extraction module;

[0055] The output end of the first splicing submodule is connected to the input end of the convolution submodule; the first output end of the convolution submodule is connected to the input end of the maximum pooling submodule, and the second output end of the convolution submodule is connected to the input end of the average pooling submodule;

[0056] The output end of the maximum pooling submodule is connected to the first input end of the second splicing submodule, the output end of the average pooling submodule is connected to the second input end of the second splicing submodule; the output end of the second splicing submodule is connected to the input end of the convolution activation submodule;

[0057] The first output end of the convolution activation submodule is connected to the input end of the third feature extraction module, and the second output end of the convolution activation submodule is connected to the input end of the fourth feature extraction module.

[0058] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0059] The present invention provides a method and system for detecting fake face videos based on multi-feature fusion, wherein a face video to be detected is frame decomposed to obtain a plurality of continuous face images to be detected, and a face fake feature is extracted from the face image to be detected through a spatial feature extraction network. The spatial feature extraction network includes a parallel first fake feature branch and a second fake feature branch, and simultaneously extracts low-frequency texture fake features and high-frequency fake features of the face image to be detected; and a cross-modal interaction module is set between the first fake feature branch and the second fake feature branch to extract inter-frame inconsistency features of the fake face video, and correspondingly adds them to the first fake feature branch and the second fake feature branch to improve the image space features extracted by the two branches, and finally obtains a fused feature map, and then inputs the plurality of fused feature maps into a temporal attention network to determine whether the face video to be detected is fake. The present invention extracts intra-frame features and inter-frame features of the fake face video in series, thereby improving the accuracy and generalization of fake face video detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0061] Figure 1 It is a flow chart of a method for detecting fake face videos based on multi-feature fusion according to the present invention;

[0062] Figure 2It is an algorithm framework diagram of the forged face video detection method based on multi-feature fusion of the present invention;

[0063] Figure 3 It is a structural schematic diagram of a cross-modal interaction module of the forged face video detection method based on multi-feature fusion of the present invention;

[0064] Figure 4 It is a schematic diagram of the structure of the forged face video detection system based on multi-feature fusion of the present invention. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] The present invention provides a method and system for detecting fake face videos based on multi-feature fusion, designs a two-stream convolutional neural network framework to extract intra-frame features based on spatial and frequency domains, uses a temporal attention Transformer network to extract inter-frame features, and takes the deep fake face video detection task as a binary classification task to train the detection network, thereby improving the accuracy and generalization of fake face video detection.

[0067] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0068] Embodiment 1

[0069] like Figure 1 As shown, this embodiment provides a method for detecting fake face videos based on multi-feature fusion, including:

[0070] Step 100, frame decomposition is performed on the face video to be detected to obtain a plurality of continuous face images to be detected.

[0071] Step 200, inputting the face image to be detected into a spatial feature extraction network to obtain a fusion feature map;

[0072] Step 300, inputting the plurality of fused feature maps into a temporal attention network to determine the type of face video; the face video type includes a normal face video and a forged face video.

[0073] Among them, the spatial feature extraction network includes a first forged feature branch, a second forged feature branch, a cross-modal interaction module and a feature fusion module; the first forged feature branch is used to perform feature extraction on the face image to be detected to obtain a first feature map; the second forged feature branch is used to perform high-pass preprocessing and feature extraction on the face image to be detected in sequence to obtain a second feature map.

[0074] The cross-modal interaction module is arranged between the first forged feature branch and the second forged feature branch, and is used to extract image feature complementary information between the first forged feature branch and the second forged feature branch, and input the image feature complementary information into the first forged feature branch and the second forged feature branch accordingly.

[0075] The input end of the feature fusion module is connected to the output end of the first forged feature branch and the output end of the second forged feature branch respectively, and the feature fusion module is used to perform feature fusion on the first feature map and the second feature map to obtain a fused feature map.

[0076] Specifically, Figure 2 As shown, in the spatial feature extraction network, the first forged feature branch includes a first feature extraction module and a third feature extraction module connected in sequence; the second forged feature branch includes a high-pass preprocessing module, a second feature extraction module and a fourth feature extraction module connected in sequence. A dual-stream architecture is constructed by the first forged feature branch and the second forged feature branch.

[0077] The first feature extraction module is used to perform RGB domain spatial feature extraction on the face image to be detected to obtain a first sub-feature map (spatial domain feature map); the high-pass preprocessing module is used to perform high-frequency noise extraction on the face image to be detected to obtain a first noise map; the second feature extraction module is used to perform feature extraction on the first noise map to obtain a second sub-feature map (frequency domain feature map); the output end of the first feature extraction module is also connected to the first input end of the cross-modal interaction module; the output end of the second feature extraction module is also connected to the second input end of the cross-modal interaction module.

[0078] Since there is some complementary information and interaction between the spatial domain feature map and the frequency domain feature map, a cross-modal interaction module is inserted between the two streams to extract complementary information and improve the extracted spatial features.

[0079] The first output end of the cross-modal interaction module is connected to the input end of the third feature extraction module; the second output end of the cross-modal interaction module is connected to the input end of the fourth feature extraction module; the cross-modal interaction module is used to interact with the first sub-feature graph and the second sub-feature graph, extract first complementary information and second complementary information, and then input the first complementary information to the third feature extraction module, and send the second complementary information to the fourth feature extraction module; the first complementary information is the distinguishing feature information of the second sub-feature graph relative to the first sub-feature graph; the second complementary information is the distinguishing feature information of the first sub-feature graph relative to the second sub-feature graph.

[0080] The third feature extraction module is used to determine the first feature map according to the first sub-feature map and the first complementary information; the fourth feature extraction module is used to determine the second feature map according to the second sub-feature map and the second complementary information. The output end of the third feature extraction module and the output end of the fourth feature extraction module are both connected to the input end of the feature fusion module. Furthermore, the feature fusion module is composed of a Transformer with a depth of 1. The extracted RGB domain features (first feature map) and frequency domain features (second feature map) are spliced ​​at the channel level, and then the fused 2048-dimensional vector is obtained through the Transformer network.

[0081] Preferably, if Figure 3 As shown, the cross-modal interaction module includes a first splicing submodule, a convolution submodule, a maximum pooling submodule, an average pooling submodule, a second splicing submodule and a convolution activation submodule.

[0082] The first input end of the first splicing submodule is connected to the output end of the first feature extraction module, and the second input end of the first splicing module is connected to the output end of the second feature extraction module. The first splicing module is used to splice two input feature maps on the channel.

[0083] The output of the first concatenation submodule is connected to the input of the convolution submodule; the first output of the convolution submodule is connected to the input of the maximum pooling submodule, and the second output of the convolution submodule is connected to the input of the average pooling submodule. The convolution submodule includes a 1x1 two-dimensional convolution, a BN layer, and a ReLU activation function in sequence to compress the number of channels of the feature map by half.

[0084] The output end of the maximum pooling submodule is connected to the first input end of the second splicing submodule, and the output end of the average pooling submodule is connected to the second input end of the second splicing submodule; the output end of the second splicing submodule is connected to the input end of the convolution activation submodule. The first output end of the convolution activation submodule is connected to the input end of the third feature extraction module, and the second output end of the convolution activation submodule is connected to the input end of the fourth feature extraction module. The maximum pooling submodule and the average pooling submodule perform maximum pooling and average pooling on the feature map respectively to obtain two single-channel feature maps, and then splice them to obtain two-channel feature maps. The convolution activation module includes 3x3 two-dimensional convolutions and Sigmoid activation functions connected in sequence to divide the feature map spliced ​​by the second splicing submodule into two single-channel feature maps and pass them to the main extraction network respectively.

[0085] The training process of the above-mentioned spatial feature extraction network specifically includes:

[0086] According to the training set and the loss function, the calibrated deep convolutional network is trained to obtain a spatial feature extraction network; the training set includes an initial face image and label information corresponding to the initial face image; the label information is the facial features of the face in the initial face image; the calibrated deep convolutional network includes a first calibrated forged feature branch, a second calibrated forged feature branch, a calibrated cross-modal interaction module, a calibrated feature fusion module and a calibrated classification module.

[0087] The connection method of the first calibrated forged feature branch, the second calibrated forged feature branch, the calibrated cross-modal interaction module, and the calibrated feature fusion module is the same as the connection method of the first forged feature branch, the second forged feature branch, the cross-modal interaction module, and the feature fusion module corresponding to the above text, and the output end of the calibration fusion module is connected to the input end of the calibration classification module; the output end of the calibration classification module is used to output the facial features of the initial face image.

[0088] Among them, the cross entropy function is used as the loss function for training, and the cross entropy loss function is:

[0089]

[0090] Among them, L represents the cross entropy loss value calculated according to the loss function, N represents the number of initial face images in the training set, and y i Represents the label information corresponding to the i-th initial face image. When the label information corresponding to the i-th initial face image is positive, y i =1; when the label information corresponding to the i-th initial face image is negative, y i =0; p iIndicates the probability that the label information of the i-th initial face image is a positive class.

[0091] After the above network training is completed, the calibration classification module is discarded to obtain a usable spatial feature extraction network. Then, based on the above spatial feature extraction network, the attention Transformer module and the classification module based on the multi-layer perceptron are connected in sequence, and the cross entropy function is still used as the loss function to train the identification network based on temporal features. At this time, the single sample of the training set is multiple frames of images in the same video numbered in chronological order.

[0092] After the spatial feature extraction network and the temporal attention network are trained, the test process of the fake face detection network based on multi-feature fusion is as follows:

[0093] 1) Decompose the video to be tested into frames to obtain fake / real images.

[0094] 2) The fake / real image is used as the input of the RGB domain spatial feature extraction network flow to obtain the first sub-feature map.

[0095] 3) At the same time, a high-pass filter rich steganalysis model (SRM) is used to obtain the high-frequency noise of the forged / real image to obtain a first noise map, which is used as the input of the second feature extraction module to obtain a second sub-feature map.

[0096] 4) Feed the first-stage features of the dual streams (the first sub-feature map and the second sub-feature map) into the cross-modal interaction module at the same time, so that the RGB domain features and high-frequency features can fully interact and extract the complementary features between them. The complementary features are added back to the previously input features, and then passed through their respective feature extraction sub-networks, and the cross-modal interaction sub-module is repeatedly used three times to fully extract the complementary features in the spatial domain and the frequency domain, and finally the first feature map and the second feature map are obtained.

[0097] 5) The extracted RGB domain features (first feature map) and frequency domain spatial features (second feature map) are combined through a feature fusion module to obtain fused features.

[0098] 6) Each ordered video frame sample is simultaneously passed through the spatial feature extraction network of the above steps 1)-5) to obtain a spatial feature sequence.

[0099] 7) The spatial feature sequence is passed through the temporal attention Transformer network to encode the temporal forged features and obtain the spatiotemporal inconsistent forged features. The forged features are then classified into two categories through the classification layer to determine whether the face video is forged.

[0100] Specifically, in step 4) above, in terms of repeatedly using three cross-modal interaction sub-modules to fully extract complementary features in the spatial domain and the frequency domain, the cross-modal interaction module specifically includes a first interaction sub-module, a second interaction sub-module and a third interaction sub-module; the first forged feature branch also includes a first feature extraction sub-module and a third feature extraction sub-module; the second forged feature branch also includes a second feature extraction sub-module and a fourth feature extraction sub-module.

[0101] The first input end of the first interaction submodule is connected to the output end of the first feature extraction module, and the first output end of the first interaction submodule is connected to the input end of the first feature extraction submodule. The second input end of the first interaction submodule is connected to the output end of the second feature extraction module; the second output end of the first interaction submodule is connected to the input end of the second feature extraction submodule.

[0102] The first input end of the second interaction submodule is connected to the output end of the first feature extraction submodule, and the first output end of the second interaction submodule is connected to the input end of the third feature extraction submodule. The second input end of the second interaction submodule is connected to the output end of the second feature extraction submodule; the second output end of the second interaction submodule is connected to the input end of the fourth feature extraction submodule.

[0103] The first input end of the third interaction submodule is connected to the output end of the third feature extraction submodule, and the first output end of the third interaction submodule is connected to the input end of the third feature extraction module. The second input end of the third interaction submodule is connected to the output end of the fourth feature extraction submodule; the second output end of the third interaction submodule is connected to the input end of the fourth feature extraction module.

[0104] The above image detection model is verified below:

[0105] The public fake face image dataset FF++ is used to train and test the model. FF++ includes four face fake methods: Deepfakes, Face2Face, FaceSwap, and NeuralTextures. Each fake method corresponds to 1,000 fake videos. 720 videos are used for training, 140 videos are used for verification, and 140 videos are used for testing. FF++ has three compressed versions: original video raw, lightly compressed video c23, and heavily compressed video c40.

[0106] The intra-frame feature extraction framework uses Xception as the backbone network. Both Xception and Transformer are initialized with model parameters pre-trained on ImageNet. The DLIB library is used for face extraction and alignment, and then the aligned face images are scaled to 256x256. Then, a 3090 GPU is used to train the model. The optimization method uses the Adam method, the learning rate is initially set to 0.0002, the weight decay is set to 5e-4, the batch size is set to 32, and the total training rounds are 50 cycles.

[0107] The model was trained according to the above training method, and then the model was tested. The results are shown in Tables 1 and 2 below:

[0108] Table 1 Detection performance on FF++

[0109]

[0110] Table 2 Generalization ability table on FF++

[0111]

[0112] As can be seen from Table 1, when the method provided by the present invention is used for forged face video detection, the accuracy on FF++ reaches the current advanced level in the industry. This is because the present invention uses the network to serially extract intra-frame features and inter-frame features.

[0113] As can be seen from Table 2, the detection performance of the method provided by the present invention on forged videos that have not been seen during training has also reached the advanced level in the industry. This is because the present invention extracts features in the spatial domain and frequency domain at the same time, and uses a fusion module based on the self-attention mechanism to fuse the two intra-frame features. Furthermore, a cross-modal interaction module based on spatial attention is used to extract complementary features in the spatial domain and frequency domain, and the combination of the above mechanisms greatly improves the generalization ability of the model.

[0114] Embodiment 2

[0115] like Figure 4 As shown, this embodiment provides a forged face video detection system based on multi-feature fusion, including:

[0116] The video decomposition unit 101 is used to decompose the video of the face to be detected into frames to obtain a plurality of continuous images of the face to be detected.

[0117] The feature fusion unit 201 is used to input the face image to be detected into a spatial feature extraction network to obtain a fused feature map.

[0118] The video type determination unit 301 is used to input the multiple fused feature maps into the temporal attention network in sequence to determine the type of face video; the face video type includes normal face video and forged face video.

[0119] Among them, the spatial feature extraction network includes a first forged feature branch, a second forged feature branch, a cross-modal interaction module and a feature fusion module; the first forged feature branch is used to perform feature extraction on the face image to be detected to obtain a first feature map; the second forged feature branch is used to perform high-pass preprocessing and feature extraction on the face image to be detected in sequence to obtain a second feature map.

[0120] The cross-modal interaction module is arranged between the first forged feature branch and the second forged feature branch, and is used to extract image feature complementary information between the first forged feature branch and the second forged feature branch, and input the image feature complementary information into the first forged feature branch and the second forged feature branch accordingly.

[0121] The input end of the feature fusion module is connected to the output end of the first forged feature branch and the output end of the second forged feature branch respectively, and the feature fusion module is used to perform feature fusion on the first feature map and the second feature map to obtain a fused feature map.

[0122] The first forged feature branch includes a first feature extraction module and a third feature extraction module connected in sequence; the second forged feature branch includes a high-pass preprocessing module, a second feature extraction module and a fourth feature extraction module connected in sequence; the first feature extraction module is used to perform RGB domain spatial feature extraction on the face image to be detected to obtain a first sub-feature map; the high-pass preprocessing module is used to perform high-frequency noise extraction on the face image to be detected to obtain a first noise map; the second feature extraction module is used to perform feature extraction on the first noise map to obtain a second sub-feature map.

[0123] The output end of the first feature extraction module is also connected to the first input end of the cross-modal interaction module; the output end of the second feature extraction module is also connected to the second input end of the cross-modal interaction module. The first output end of the cross-modal interaction module is connected to the input end of the third feature extraction module; the second output end of the cross-modal interaction module is connected to the input end of the fourth feature extraction module; the cross-modal interaction module is used to interact with the first sub-feature graph and the second sub-feature graph, extract the first complementary information and the second complementary information, and then input the first complementary information to the third feature extraction module, and send the second complementary information to the fourth feature extraction module; the first complementary information is the distinguishing feature information of the second sub-feature graph relative to the first sub-feature graph; the second complementary information is the distinguishing feature information of the first sub-feature graph relative to the second sub-feature graph.

[0124] The third feature extraction module is used to determine the first feature map according to the first sub-feature map and the first complementary information; the fourth feature extraction module is used to determine the second feature map according to the second sub-feature map and the second complementary information. The output end of the third feature extraction module and the output end of the fourth feature extraction module are both connected to the input end of the feature fusion module.

[0125] Specifically, the cross-modal interaction module includes a first splicing submodule, a convolution submodule, a maximum pooling submodule, an average pooling submodule, a second splicing submodule and a convolution activation submodule.

[0126] The first input end of the first splicing submodule is connected to the output end of the first feature extraction module, and the second input end of the first splicing module is connected to the output end of the second feature extraction module. The output end of the first splicing submodule is connected to the input end of the convolution submodule; the first output end of the convolution submodule is connected to the input end of the maximum pooling submodule, and the second output end of the convolution submodule is connected to the input end of the average pooling submodule.

[0127] The output end of the maximum pooling submodule is connected to the first input end of the second splicing submodule, the output end of the average pooling submodule is connected to the second input end of the second splicing submodule; the output end of the second splicing submodule is connected to the input end of the convolution activation submodule. The first output end of the convolution activation submodule is connected to the input end of the third feature extraction module, and the second output end of the convolution activation submodule is connected to the input end of the fourth feature extraction module.

[0128] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0129] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for detecting fake face videos based on multi-feature fusion, characterized in that: The forged face video detection method comprises: Decompose the video of the face to be detected into frames to obtain multiple continuous images of the face to be detected; Inputting the face image to be detected into a spatial feature extraction network to obtain a fused feature map; Inputting the plurality of fused feature maps into a temporal attention network to determine the type of face video; the type of face video includes a normal face video and a forged face video; The spatial feature extraction network includes a first forged feature branch, a second forged feature branch, a cross-modal interaction module and a feature fusion module; the first forged feature branch is used to perform feature extraction on the face image to be detected to obtain a first feature map; the second forged feature branch is used to perform high-pass preprocessing and feature extraction on the face image to be detected in sequence to obtain a second feature map; The cross-modal interaction module is arranged between the first forged feature branch and the second forged feature branch, and is used to extract image feature complementary information between the first forged feature branch and the second forged feature branch, and input the image feature complementary information to the first forged feature branch and the second forged feature branch accordingly; The input end of the feature fusion module is connected to the output end of the first forged feature branch and the output end of the second forged feature branch respectively, and the feature fusion module is used to perform feature fusion on the first feature map and the second feature map to obtain a fused feature map.

2. The method for detecting fake face videos based on multi-feature fusion according to claim 1, characterized in that: The first forged feature branch includes a first feature extraction module and a third feature extraction module connected in sequence; the second forged feature branch includes a high-pass preprocessing module, a second feature extraction module and a fourth feature extraction module connected in sequence; The first feature extraction module is used to perform RGB domain spatial feature extraction on the face image to be detected to obtain a first sub-feature map; The high-pass preprocessing module is used to extract high-frequency noise from the face image to be detected to obtain a first noise map; the second feature extraction module is used to extract features from the first noise map to obtain a second sub-feature map; The output end of the first feature extraction module is also connected to the first input end of the cross-modal interaction module; the output end of the second feature extraction module is also connected to the second input end of the cross-modal interaction module; The first output end of the cross-modal interaction module is connected to the input end of the third feature extraction module; the second output end of the cross-modal interaction module is connected to the input end of the fourth feature extraction module; the cross-modal interaction module is used to interact with the first sub-feature graph and the second sub-feature graph, extract the first complementary information and the second complementary information, and then input the first complementary information to the third feature extraction module, and send the second complementary information to the fourth feature extraction module; the first complementary information is the distinguishing feature information of the second sub-feature graph relative to the first sub-feature graph; the second complementary information is the distinguishing feature information of the first sub-feature graph relative to the second sub-feature graph; The third feature extraction module is used to determine a first feature map according to the first sub-feature map and the first complementary information; the fourth feature extraction module is used to determine a second feature map according to the second sub-feature map and the second complementary information; The output end of the third feature extraction module and the output end of the fourth feature extraction module are both connected to the input end of the feature fusion module.

3. The method for detecting fake face videos based on multi-feature fusion according to claim 2, characterized in that: The cross-modal interaction module includes a first splicing submodule, a convolution submodule, a maximum pooling submodule, an average pooling submodule, a second splicing submodule and a convolution activation submodule; A first input end of the first splicing submodule is connected to an output end of the first feature extraction module, and a second input end of the first splicing submodule is connected to an output end of the second feature extraction module; The output end of the first splicing submodule is connected to the input end of the convolution submodule; the first output end of the convolution submodule is connected to the input end of the maximum pooling submodule, and the second output end of the convolution submodule is connected to the input end of the average pooling submodule; The output end of the maximum pooling submodule is connected to the first input end of the second splicing submodule, the output end of the average pooling submodule is connected to the second input end of the second splicing submodule; the output end of the second splicing submodule is connected to the input end of the convolution activation submodule; The first output end of the convolution activation submodule is connected to the input end of the third feature extraction module, and the second output end of the convolution activation submodule is connected to the input end of the fourth feature extraction module.

4. The method for detecting fake face videos based on multi-feature fusion according to claim 1, characterized in that: The training process of the spatial feature extraction network specifically includes: According to the training set and the loss function, the calibration deep convolutional network is trained to obtain a spatial feature extraction network; the training set includes an initial face image and label information corresponding to the initial face image; the label information is the facial features of the face in the initial face image; the calibration deep convolutional network includes a first calibration forged feature branch, a second calibration forged feature branch, a calibration cross-modal interaction module, a calibration feature fusion module and a calibration classification module; The input end of the first calibration forged feature branch is used to input the initial face image; the output end of the first calibration forged feature branch is connected to the first input end of the calibration feature fusion module; The input end of the second calibration forged feature branch is used to input the initial face image; the output end of the second calibration forged feature branch is connected to the second input end of the calibration feature fusion module; The calibration cross-modal interaction module is arranged between the first calibration forged feature branch and the second calibration forged feature branch; The output end of the calibration feature fusion module is connected to the input end of the calibration classification module; the output end of the calibration classification module is used to output the facial features of the initial face image.

5. The method for detecting fake face videos based on multi-feature fusion according to claim 2, characterized in that: The cross-modal interaction module includes a first interaction submodule, a second interaction submodule and a third interaction submodule; the first forged feature branch also includes a first feature extraction submodule and a third feature extraction submodule; the second forged feature branch also includes a second feature extraction submodule and a fourth feature extraction submodule; The first input end of the first interaction submodule is connected to the output end of the first feature extraction module, and the first output end of the first interaction submodule is connected to the input end of the first feature extraction submodule; The second input end of the first interaction submodule is connected to the output end of the second feature extraction module; the second output end of the first interaction submodule is connected to the input end of the second feature extraction submodule; The first input end of the second interaction submodule is connected to the output end of the first feature extraction submodule, and the first output end of the second interaction submodule is connected to the input end of the third feature extraction submodule; The second input end of the second interaction submodule is connected to the output end of the second feature extraction submodule; the second output end of the second interaction submodule is connected to the input end of the fourth feature extraction submodule; The first input end of the third interaction submodule is connected to the output end of the third feature extraction submodule, and the first output end of the third interaction submodule is connected to the input end of the third feature extraction module; The second input end of the third interaction submodule is connected to the output end of the fourth feature extraction submodule; the second output end of the third interaction submodule is connected to the input end of the fourth feature extraction module.

6. The method for detecting fake face videos based on multi-feature fusion according to claim 1, characterized in that: The temporal attention network includes an attention Transformer module and a classification module based on a multi-layer perceptron connected in sequence.

7. A forged face video detection system based on multi-feature fusion, characterized in that: The forged face video detection system comprises: A video decomposition unit is used to decompose the video of the face to be detected into frames to obtain multiple continuous images of the face to be detected; A feature fusion unit, used for inputting the face image to be detected into a spatial feature extraction network to obtain a fused feature map; A video type determination unit, used for inputting the plurality of fused feature maps into a temporal attention network to determine the type of face video; the face video type includes a normal face video and a forged face video; The spatial feature extraction network includes a first forged feature branch, a second forged feature branch, a cross-modal interaction module and a feature fusion module; the first forged feature branch is used to perform feature extraction on the face image to be detected to obtain a first feature map; the second forged feature branch is used to perform high-pass preprocessing and feature extraction on the face image to be detected in sequence to obtain a second feature map; The cross-modal interaction module is arranged between the first forged feature branch and the second forged feature branch, and is used to extract image feature complementary information between the first forged feature branch and the second forged feature branch, and input the image feature complementary information to the first forged feature branch and the second forged feature branch accordingly; The input end of the feature fusion module is connected to the output end of the first forged feature branch and the output end of the second forged feature branch respectively, and the feature fusion module is used to perform feature fusion on the first feature map and the second feature map to obtain a fused feature map.

8. The forged face video detection system based on multi-feature fusion according to claim 7 is characterized in that: The first forged feature branch includes a first feature extraction module and a third feature extraction module connected in sequence; the second forged feature branch includes a high-pass preprocessing module, a second feature extraction module and a fourth feature extraction module connected in sequence; The first feature extraction module is used to perform RGB domain spatial feature extraction on the face image to be detected to obtain a first sub-feature map; The high-pass preprocessing module is used to extract high-frequency noise from the face image to be detected to obtain a first noise map; the second feature extraction module is used to extract features from the first noise map to obtain a second sub-feature map; The output end of the first feature extraction module is also connected to the first input end of the cross-modal interaction module; the output end of the second feature extraction module is also connected to the second input end of the cross-modal interaction module; The first output end of the cross-modal interaction module is connected to the input end of the third feature extraction module; the second output end of the cross-modal interaction module is connected to the input end of the fourth feature extraction module; the cross-modal interaction module is used to interact with the first sub-feature graph and the second sub-feature graph, extract the first complementary information and the second complementary information, and then input the first complementary information to the third feature extraction module, and send the second complementary information to the fourth feature extraction module; the first complementary information is the distinguishing feature information of the second sub-feature graph relative to the first sub-feature graph; the second complementary information is the distinguishing feature information of the first sub-feature graph relative to the second sub-feature graph; The third feature extraction module is used to determine a first feature map according to the first sub-feature map and the first complementary information; the fourth feature extraction module is used to determine a second feature map according to the second sub-feature map and the second complementary information; The output end of the third feature extraction module and the output end of the fourth feature extraction module are both connected to the input end of the feature fusion module.

9. The forged face video detection system based on multi-feature fusion according to claim 8 is characterized in that: The cross-modal interaction module includes a first splicing submodule, a convolution submodule, a maximum pooling submodule, an average pooling submodule, a second splicing submodule and a convolution activation submodule; A first input end of the first splicing submodule is connected to an output end of the first feature extraction module, and a second input end of the first splicing submodule is connected to an output end of the second feature extraction module; The output end of the first splicing submodule is connected to the input end of the convolution submodule; the first output end of the convolution submodule is connected to the input end of the maximum pooling submodule, and the second output end of the convolution submodule is connected to the input end of the average pooling submodule; The output end of the maximum pooling submodule is connected to the first input end of the second splicing submodule, the output end of the average pooling submodule is connected to the second input end of the second splicing submodule; the output end of the second splicing submodule is connected to the input end of the convolution activation submodule; The first output end of the convolution activation submodule is connected to the input end of the third feature extraction module, and the second output end of the convolution activation submodule is connected to the input end of the fourth feature extraction module.

Images