Multi-scale feature fusion tampering video detection method, system, medium and equipment

Abstract

The present application relates to a kind of multiscale feature fusion tampered video detection method, system, medium and equipment, comprising: obtaining video frame by video stream, extracting face region image in each video frame;Face region image is carried out data enhancement;The enhanced face region image is input into Xception network to extract the spatial tampering feature vector of face region image;Spatial tampering feature vector is input into FPN and LSTM network respectively to extract corresponding feature vector;The feature vector extracted by FPN and LSTM network is multidimensional feature fusion, obtains fusion feature vector;Fusion feature vector is input into SVM classifier, and the output result of SVM classifier can characterize whether video stream is tampered with.It not only greatly improves the accuracy, robustness and generalization of model when detecting, but also improves the detection ability of model for different compression ratio, different scale size image to some extent.

Description

TECHNICAL FIELD

[0001] The present application relates to a multi-scale feature fusion tampered video detection method, system, medium and equipment, belonging to the technical field of video detection, in particular to the technical field of tampered video detection based on deep learning algorithm. BACKGROUND

[0002] With the continuous development of artificial intelligence technology, face tampering technology based on deep learning is becoming more mature. A kind of face tampering method represented by Deepfake is widely spread on the Internet, which has the characteristics of low learning cost, realistic tampering effect and strong deception, and has caused a certain degree of negative impact on society.

[0003] At present, the detection of tampered videos can be briefly divided into two categories: detection methods based on hand-crafted features and detection methods based on deep learning. The detection method based on hand-crafted features often needs to construct one or several effective feature extractors according to the differences between tampered videos and real videos. When the feature does not exist in the video to be detected, the detection accuracy will be greatly reduced, so the generalization ability of this method has great limitations

[0004] The detection method based on deep learning often learns the features of the input sample in the spatial domain through convolutional neural network, or learns the time domain features of the sample through LSTM and other time sequence networks, or maps the input to the frequency domain through FFT and then extracts the frequency domain features. However, these methods have the defects of poor robustness of cross-library detection effect, serious overfitting phenomenon, insufficient model generalization ability, and the detection ability of different scale images decreases; in addition, many tampered video detection algorithms based on multi-feature fusion also have the shortcoming that different features are directly fused with equal weight. SUMMARY

[0005] In view of the above problems, the purpose of the present application is to provide a multi-scale feature fusion tampered video detection method, system, medium and equipment, which not only greatly improves the accuracy, robustness and generalization of the model during detection, but also improves the detection ability of the model for different compression rates and different scale images to a certain extent.

[0006] To achieve the above objectives, the present invention proposes the following technical solutions: a multi-scale feature fusion tampering video detection method, comprising: obtaining video frames through a video stream, extracting a face area image in each video frame; performing data enhancement on the face area image; inputting the enhanced face area image into an Xception network to extract a spatial tampering feature vector of the face area image; inputting the spatial tampering feature vector into an FPN and LSTM network respectively to extract corresponding feature vectors; performing multi-dimensional feature fusion on the feature vectors extracted by the FPN and LSTM networks to obtain a fused feature vector; and inputting the fused feature vector into an SVM classifier, wherein the output result of the SVM classifier can characterize whether the video stream has been tampered with.

[0007] Furthermore, the method for extracting face area images is as follows: use OpenCV to sample the video stream to obtain a video frame sequence {frame1, frame2,…, frameN}; perform face detection and extraction on each video frame in the video frame sequence through the MTCNN face detection algorithm, and adjust the extracted face area RGB image to a uniform size to obtain a face area image sequence {face1, face2,…, faceN}.

[0008] Furthermore, data enhancement methods include image cropping, random flipping, random addition of Gaussian noise, random addition of blur processing, random addition of affine transformation and random brightness and contrast adjustment.

[0009] Furthermore, a data iterator is constructed for the data-enhanced face area images to realize batch reading of iterative data.

[0010] Furthermore, in the Xception network, the last classification layer is replaced by a maximum pooling layer.

[0011] Furthermore, the FPN network is used to extract video tampering features in forged videos, and the LSTM network is used to learn abstract semantic information between adjacent dimensions in the feature space.

[0012] Furthermore, the method of multi-dimensional feature fusion is as follows: the feature vectors output by the FPN and LSTM networks are spliced ​​into a spliced ​​feature vector; the spliced ​​feature vector is input into the SENet network to obtain the weight feature vector of each channel; and the fused feature vector is obtained by multiplying the spliced ​​feature vector and the weight feature vector of each channel element by element.

[0013] The present invention also discloses a multi-scale feature fusion tampering video detection system, comprising: a face extraction module, used to obtain video frames through a video stream and extract a face area image in each video frame; an image enhancement module, used to perform data enhancement on the face area image; a spatial tampering feature extraction module, used to input the enhanced face area image into an Xception network to extract a spatial tampering feature vector of the face area image; a semantic feature extraction module, used to input the spatial tampering feature vector into an FPN and LSTM network respectively to extract a corresponding feature vector; a feature fusion module, used to perform multi-dimensional feature fusion on the feature vectors extracted by the FPN and LSTM networks to obtain a fused feature vector; an SVM output module, used to input the fused feature vector into an SVM classifier, and the output result of the SVM classifier can characterize whether the video stream has been tampered with.

[0014] The present invention also discloses a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions. When the instructions are executed by a computing device, the computing device executes any of the above-mentioned multi-scale feature fusion tampering video detection methods.

[0015] The present invention also discloses a computing device comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for executing any of the above-mentioned multi-scale feature fusion tampering video detection methods.

[0016] The present invention has the following advantages due to the adoption of the above technical solution:

[0017] 1. In the data preprocessing stage, the present invention uses a variety of image data enhancement methods including image cropping, random flipping, random addition of Gaussian noise, random addition of blur processing, random addition of affine transformation and random brightness and contrast adjustment, which effectively improves the robustness and generalization of the model;

[0018] 2. This invention introduces the FPN feature pyramid network to fuse features at different scales, which improves the model's detection ability for images of different compression rates and sizes to a certain extent, avoiding the problem of significant decline in cross-database detection that exists in most current mainstream models.

[0019] 3. By introducing the LSTM network, the present invention can effectively learn the relevant information between adjacent dimensions of the feature space, find the abstract connection between features, and further enhance the generalization ability of the model;

[0020] 4. The present invention avoids the direct superposition of equal weights between features by adding an attention mechanism during feature fusion. On the contrary, the network can learn the proportion of weights between different features through training, which can better complete the fusion of features, greatly improving the robustness and generalization of the model during detection, and also significantly improving the accuracy of training. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a schematic diagram of a method for detecting tampered videos using multi-scale feature fusion according to an embodiment of the present invention;

[0022] Figure 2 is a comparison chart of the video tampering detection method of the present invention and six different existing detection methods. Figure 2 (a) is the training accuracy image, Figure 2 (b) is the training loss image. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical direction of the present invention, the present invention will be described in detail through specific embodiments. However, it should be understood that the provision of specific embodiments is only for a better understanding of the present invention and they should not be construed as limitations of the present invention. In the description of the present invention, it should be understood that the terms used are for descriptive purposes only and are not to be construed as indicating or implying relative importance.

[0024] The present invention relates to a multi-scale feature fusion tampering video detection method, system, medium and device, which extracts the RGB image of the face area in the video frame, first inputs the image data into the Xception network for preliminary feature extraction; then respectively inputs the image data into the FPN network and the LSTM network to extract the image tampering information of the different scale images and the related information of the spatial adjacent dimensions, and finally performs feature classification through the SVM classifier to obtain the final detection result. The present invention simulates the noise, blur, brightness and contrast changes generated after the Deepfake tampering video is uploaded to the Internet by data enhancement means, and at the same time introduces the FPN and LSTM networks to realize the feature fusion of the Deepfake tampering information of different scales and the related information of the adjacent dimensions of the feature space. On this basis, the attention mechanism is used to realize the self-learning of the feature fusion weights. The above means significantly improve the generalization ability of the model and the robustness of cross-dataset detection. The following is a detailed description of the scheme of the present invention through several embodiments in conjunction with the accompanying drawings.

[0025] Example 1

[0026] This embodiment discloses a method for detecting video tampering by fusion of multi-scale features. Figure 1 Shown, including:

[0027] S1 obtains video frames by sampling the video stream at a set frame rate, and extracts the face area image from each video frame in the video frame sequence through the MTCNN face detection algorithm. The face area image is an RGB image.

[0028] In this example, the Celeb-DFv1 dataset was selected, which contains 408 original videos from YouTube. 795 deepfake videos were synthesized using the Deepfake tool. The real and fake video streams in the dataset were sampled at a preset frame rate k and decoded to produce video frames. The video streams were then decoded using the OpenCV VideoCapture class and all the corresponding video frames were saved in a folder named after the video file. This completes the video stream extraction process, resulting in a video frame sequence {frame1, frame2, …, frameN}.

[0029] After extracting each video frame in the video frame sequence, traverse all video folders to obtain video frames and save them through OpenCV. The MTCNN face detection algorithm integrated in the pytorch_facenet software is used to perform face detection and extraction on each video frame in the video frame sequence to obtain a face area image sequence {face1, face2, ..., faceN}. When the MTCNN face detection algorithm is instantiated, all extracted face area images are adjusted to a uniform size by adding the input parameter output_size. In this embodiment, the uniform size is preferably 224×224×3, where 224 and 3 are both pixel values. The save function in the numpy software is used to store the data of all face area images in the form of ndarray in numpy in an npy file. Similarly, all labels corresponding to the face area images are also converted into numpy ndarray format and stored in an npy file. After completion, two npy files are obtained, which store the face area images and their corresponding labels respectively.

[0030] S2 performs data enhancement on the face area image.

[0031] During the training process, in order to enhance the generalization ability of the model, image data augmentation operations are performed on the face area image sequence {face1, face2,…, faceN}. The data augmentation methods include image cropping, random flipping, random addition of Gaussian noise, random addition of blur processing, random addition of affine transformation and random brightness and contrast adjustment.

[0032] A data iterator, DataLoader, is constructed for the augmented facial region images to iterate over batches of samples and labels from the dataset. Therefore, a Deepfake_dataset class is constructed and passed to the PyTorch DataLoader class to complete the data iterator construction. The Deepfake_dataset class inherits from the PyTorch nn.datasets class and requires implementing __init__ for initialization, __len__ for obtaining the dataset size, and __getitem__ for retrieving batches of samples and labels. In __getitem__, the npy files for the facial region images and their corresponding labels are read and converted from ndarray format to PyTorch's Tensor format. The albumentations tool is then used to perform operations on the images, including cropping, random flipping, random addition of Gaussian noise, random blurring, random addition of affine transformations, and random brightness and contrast adjustments. The augmented data is then normalized to convert it to a standard Gaussian distribution. This normalization process accelerates backpropagation during training. The specific formula for data normalization is as follows:

[0033]

[0034] Where mean represents the mean of each channel, and std represents the standard deviation of each channel. Considering that the pre-trained weights of the Xception network will be used for feature extraction during formal training, in this example, the mean and std values ​​are set to [0.485, 0.456, 0.406] and [0.229, 0.224, 0.225], respectively. These values ​​were calculated during large-scale training of the Xception network on the ImageNet dataset and have good generalizability and accuracy.

[0035] S3 inputs the enhanced face area image into the Xception network to extract the spatial tampering feature vector of the face area image.

[0036] In the Xception network, the last classification layer is replaced by the maximum pooling layer so that its output size is preferably a spatial tampering feature vector of 1×1×2048, denoted as V_backbone, which is used to characterize the results of the initial feature extraction.

[0037] S4 inputs the spatial tampering feature vector into the FPN and LSTM networks respectively to extract the corresponding feature vector.

[0038] The FPN network is used to extract video tampering features in a fake video, and the LSTM network is used to learn abstract semantic information between adjacent dimensions in a feature space.

[0039] The spatial tampering feature vector V_backbone is input into the FPN feature pyramid network to extract video tampering features of different scales of images. The video tampering features include spatial traces of video tampering, such as artifacts of a face, differences in color temperature, and the like. The dimension of the vector input into the FPN network is batch_sizex1x1x2048, that is, a 1x1 pixel is used to represent the semantic information of the entire image by embedding a 2048-dimensional vector. This semantic information is low-scale semantic information. In order to obtain higher-scale semantic information, the input vector needs to be upsampled three times through convolution operation. Finally, the feature map is flattened through the view operation in PyTorch, and the upsampled feature map is mapped to a spatial dimension of 256 through a linear fully connected layer. Finally, a dropout operation is added to alleviate overfitting, and an output vector V_fpn is obtained, which has a size of 1x256, where 256 is the size of the hidden layer in the FPN network.

[0040] The spatial tampering feature vector V_backbone is input into the LSTM long short-term memory network to extract abstract semantic information between adjacent dimensions in a feature space. The size of the spatial tampering feature vector V_backbone is batch_sizex1x1x2048. In this embodiment, the spatial tampering feature vector V_backbone is expanded according to the feature dimension through the view operation in PyTorch, and the 2048-dimensional features are divided into 32 feature subsets, each containing a 64-dimensional feature vector. The size of the divided feature is batch_sizex32x64. Then it is input into the LSTM network with a sequence length of 32 and an input vector length of 64, and an output vector V_lstm is obtained, which has a size of 1x256, where 256 is the length of the encoding vector in the hidden layer of the LSTM network.

[0041] S5 multi-dimensional feature fusion is performed on the feature vectors extracted by the FPN and LSTM networks to obtain a fused feature vector.

[0042] The method of multi-dimensional feature fusion in this embodiment is as follows:

[0043] The feature vectors output by the FPN and LSTM networks are spliced into a spliced feature vector, denoted as V_feature, which has a size of 1x512;

[0044] The spliced feature vector V_feature is input into the SE network to obtain a weight feature vector of each channel of the spliced feature vector, denoted as SE_weight, which has a size of 1x512;

[0045] The fused feature vector is obtained by multiplying the concatenated feature vector and the weight feature vector of each channel element by element, i.e. V_feature⊙SE_weight.

[0046] S6 inputs the fused feature vector into a pre-trained SVM classifier, and the output result of the SVM classifier can characterize whether the video stream has been tampered with.

[0047] In this embodiment, the initial learning rate of the SVM classifier is set to 0.0001, the warm-up learning rate is set to 0.1, and the learning rate decay strategy uses cosine annealing decay to dynamically adjust the learning rate, making the learning rate decay smoother.

[0048] The SVM classifier optimizer uses AdamW, the decay factor is set to 0.00005, the batch_size is set to 128, the loss function uses cross-entropy, and 10 epoches are trained on NVIDIA's TITANX graphics card. The output results are visualized using the visdom module to monitor the changes in various indicators during training.

[0049] After the training is completed, the model weight parameters are saved, the features extracted by the deep neural network are loaded and passed to the SVM classifier for training 50 epoches. After the SVM training is completed, the new forged video can be detected to determine whether the video stream to be tested has been tampered with by Deepfake.

[0050] In order to verify the performance of the model, under the premise of controlling the variables, the following different models are trained respectively: Xcpetion, LSTM, Vision-Transformer, ResNet and this method. After training 10 epoches, the training results are output through the visdom module. Figure 2 As shown, Figure 2 (a) is the training accuracy image, Figure 2 (b) is the training loss graph. In the training accuracy graph, the proposed method shows the accuracy curve of the corresponding method of this embodiment as a function of the number of training generations. This shows that compared to the other four methods, the method of this embodiment achieves higher accuracy at the beginning of training. Furthermore, as the number of generations increases, it still achieves the highest accuracy of the four methods. In the training loss graph, compared to the other methods, the method of this embodiment still achieves the lowest training loss. Therefore, this embodiment has better model performance than the four existing verification methods.

[0051] Example 2

[0052] Based on the same inventive concept, this embodiment discloses a multi-scale feature fusion tampering video detection system, including:

[0053] a face extraction module configured to obtain video frames from the video stream and extract face region images in each video frame;

[0054] an image enhancement module configured to perform data enhancement on the face region images;

[0055] a spatial tampering feature extraction module configured to input the enhanced face region images into an Xception network to extract spatial tampering feature vectors of the face region images;

[0056] a semantic feature extraction module configured to input the spatial tampering feature vectors into FPN and LSTM networks respectively to extract corresponding feature vectors;

[0057] a feature fusion module configured to perform multi-dimensional feature fusion on the feature vectors extracted by the FPN and LSTM networks to obtain a fusion feature vector;

[0058] an SVM output module configured to input the fusion feature vector into an SVM classifier, and an output result of the SVM classifier is capable of representing whether the video stream is tampered.

[0059] Embodiment Three

[0060] Based on the same inventive concept, the embodiment discloses a computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by a computing device, cause the computing device to perform the multi-scale feature fusion tampered video detection method according to any one of the above embodiments.

[0061] Embodiment Four

[0062] Based on the same inventive concept, the embodiment discloses a computing device, including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing the multi-scale feature fusion tampered video detection method according to any one of the above embodiments.

[0063] Those skilled in the art will understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0064] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0065] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0066] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the specific embodiments of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be included within the scope of protection of the claims of the present invention. The above content is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present application, which should be included within the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A multi-scale feature fusion tampering video detection method, characterized in that: include: Obtaining video frames through a video stream, and extracting a face region image in each of the video frames; Data enhancement is performed on the face area image; the data enhancement method includes image cropping, random flipping, random addition of Gaussian noise, random addition of blur processing, random addition of affine transformation and random brightness and contrast adjustment; the enhanced data is normalized, and the specific formula for data normalization is as follows: in, represents the mean value of each channel, Indicates the standard deviation of each channel; Inputting the enhanced face region image into the Xception network to extract the spatial tampering feature vector of the face region image; The spatial tampering feature vector is input into the FPN and LSTM networks respectively to extract the corresponding feature vector: The spatial tampering feature vector is input into the FPN feature pyramid network to extract the video tampering features of images of different scales. The semantic information of the entire image is represented by embedding the spatial tampering feature vector. The spatial tampering feature vector is upsampled three times through the convolution operation. Finally, the feature map is flattened through the view operation in PyTorch, and the upsampled feature map is mapped to a spatial dimension of 256 through the linear fully connected layer. Finally, the drop out operation is added to alleviate overfitting and obtain the output vector. ; The spatial tampering feature vector is input into the LSTM long short-term memory network to extract the abstract semantic information between adjacent dimensions in the feature space. The spatial tampering feature vector is expanded according to the feature dimension through the view operation in PyTorch, and then the expanded feature vector is input into the LSTM network to obtain the output vector ; Perform multi-dimensional feature fusion on the feature vectors extracted by FPN and LSTM networks to obtain a fused feature vector; The fused feature vector is input into an SVM classifier, and the output result of the SVM classifier can characterize whether the video stream has been tampered with.

2. The multi-scale feature fusion tampering video detection method according to claim 1, characterized in that: The method for extracting the face area image is: using OpenCV to sample the video stream to obtain a video frame sequence ; Use the MTCNN face detection algorithm to detect and extract faces in each video frame in the video frame sequence, and adjust the extracted face area RGB image to a uniform size to obtain a face area image sequence .

3. The multi-scale feature fusion tampering video detection method according to claim 2, characterized in that: A data iterator is constructed for the face region image enhanced by the data to realize batch reading of iterative data.

4. The multi-scale feature fusion tampering video detection method according to claim 1, characterized in that: In the Xception network, the last classification layer is replaced by a maximum pooling layer.

5. The multi-scale feature fusion tampering video detection method according to claim 1, characterized in that: The method of multi-dimensional feature fusion is: Concatenate the feature vectors output by the FPN and LSTM networks into a concatenated feature vector; Input the concatenated feature vector into the SENet network to obtain the weight feature vector of each channel; The fused feature vector is obtained by multiplying the concatenated feature vector and the weight feature vector of each channel element by element.

6. A multi-scale feature fusion tampering video detection system, characterized by: include: A face extraction module is used to obtain video frames through a video stream and extract a face area image in each video frame; An image enhancement module, configured to perform data enhancement on the face area image; The data enhancement method includes image cropping, random flipping, random addition of Gaussian noise, random addition of blur processing, random addition of affine transformation and random brightness and contrast adjustment; the enhanced data is normalized, and the specific formula for data normalization is as follows: in, represents the mean value of each channel, Indicates the standard deviation of each channel; A spatial tampering feature extraction module is used to input the enhanced face area image into the Xception network to extract the spatial tampering feature vector of the face area image; A semantic feature extraction module is used to input the spatial tampering feature vector into the FPN and LSTM networks respectively to extract the corresponding feature vector; The spatial tampering feature vector is input into the FPN feature pyramid network to extract the video tampering features of images of different scales. The semantic information of the entire image is represented by embedding the spatial tampering feature vector. The spatial tampering feature vector is upsampled three times through the convolution operation. Finally, the feature map is flattened through the view operation in PyTorch, and the upsampled feature map is mapped to a spatial dimension of 256 through the linear fully connected layer. Finally, the drop out operation is added to alleviate overfitting and obtain the output vector. ; The spatial tampering feature vector is input into the LSTM long short-term memory network to extract the abstract semantic information between adjacent dimensions in the feature space. The spatial tampering feature vector is expanded according to the feature dimension through the view operation in PyTorch, and then the expanded feature vector is input into the LSTM network to obtain the output vector ; The feature fusion module is used to perform multi-dimensional feature fusion on the feature vectors extracted by the FPN and LSTM networks to obtain a fused feature vector; The SVM output module is used to input the fused feature vector into an SVM classifier, and the output result of the SVM classifier can characterize whether the video stream has been tampered with.

7. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform the multi-scale feature fusion tampering video detection method according to any one of claims 1 to 5.

8. A computing device, characterized in that include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a method for executing the multi-scale feature fusion tampering video detection method according to any one of claims 1 to 5.

Images