Deepfake video detection segmentation identification system, terminal and storage medium
The deepfake video detection system, which utilizes multi-scale feature fusion and important region localization, addresses the low accuracy of existing technologies across compression rates and datasets, achieving high-accuracy detection under various conditions.
Patent Information
- Application Number
- CN202211639132.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-20
- Publication Date
- 2026-01-13
- Estimated Expiration
- 2042-12-20
AI Technical Summary
Existing deepfake video detection technologies have low accuracy when tested across compression rates or datasets, are difficult to generalize, and are easily affected by artifact changes caused by video compression and unclear areas in the real video.
By learning features at multiple scales, and employing modules such as data preprocessing, feature extraction, important region localization, local random elimination, and multi-scale feature fusion, the model's detection capabilities on different compression rates or datasets are improved.
This improved the model's generalization performance across different compression rates and datasets, ensuring high accuracy when dealing with videos of varying resolutions and compression rates.
Smart Images
Figure CN115909172B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the field of computer vision, and particularly relates to a deepfake video detection segmentation and recognition system, a terminal and a storage medium. BACKGROUND
[0002] With the development of deep learning algorithms and the improvement of various hardware computing capabilities, a series of generation models represented by generative adversarial networks have emerged like mushrooms after the rain. This technology has been gradually applied to many fields in life, such as image style transfer, image high-resolution reconstruction, picture repair and 3D printing.
[0003] However, in recent years, this technology has been misused, and deepfake technology is one of the most controversial applications. The term "deepfake" is translated from the English word "deepfake", which is a combination of the words "deep learning" and "fake", meaning a false image or video generated by a deep learning model. Some software that provides face-swapping video generation function has lowered the threshold for using this technology, which means that anyone can replace the characters in a video with other specified objects at a very low cost. The emergence of these software has raised concerns about this technology and the attention of the scientific community. Since then, deepfake video generation technology and deepfake video detection technology have been continuously developing, and the two technologies present a state of game. The former aims to generate videos with better quality and more realistic videos, while the latter aims to accurately distinguish whether a video is recorded by a real physical device or generated by an algorithm.
[0004] One of the difficulties of deepfake video detection is that a model trained on dataset A will have a significantly lower accuracy on dataset B. This is mainly because the area and size of artifacts in videos generated by different generation algorithms have large differences. In addition, even if the same dataset as the training set is used for testing, the performance of the model will also decrease significantly if the videos in the test set are compressed to a certain extent. This is mainly because some information is lost when the video is compressed, which will directly lead to changes in the size of the artifact area in the fake video, making it more difficult for the detection model to capture this feature. In addition, compressed real videos may have some unclear or discontinuous local areas. These dirty information generated during video processing may cause the detection model to mistakenly identify real videos as deepfake videos. SUMMARY
[0005] In view of the problem that existing deep fake video detection technologies have low generalization performance, the application provides a deep fake video detection segmentation and identification system, a terminal and a storage medium, which learn features of multiple scales to improve the accuracy of the model when testing across compression rates or across data sets.
[0006] The technical solutions adopted by the application are as follows:
[0007] A deep fake video detection system comprises:
[0008] A data preprocessing module is configured to extract a face region from a video, wherein the input of the data preprocessing module is a video file, and the output of the data preprocessing module is a plurality of images containing the face region and enhanced by adding Gaussian noise and simulating different video quality compression changes in an actual scene.
[0009] A feature extraction module is configured to extract features from a global or local region of a face, wherein the input of the feature extraction module is a complete face image or a local region, and the output of the feature extraction module is a feature map with a size of 14*14.
[0010] An important region positioning module is configured to extract regions with the most discriminative significance from samples in a training process, i.e., to locate a plurality of most important regions, and comprises a non-maximum suppression (NMS) screening layer. Most artifacts in deep fake videos are distributed around the features of a face, the size of artifacts around eyes is limited by the size of the eyes, and the nose and the chin, as relatively three-dimensional regions of the face, are prone to generate large-size artifacts. Different sizes of sliding windows are selected in the positioning process, and the distribution and scale characteristics of the artifacts in deep fake videos in an actual scene are simulated. The input of the important region positioning module is the 14*14 feature map extracted by the feature extraction module from the face image, and the output of the important region positioning module is the vertex coordinates and the feature map of the extracted regions. Different sizes of sliding windows are selected in the positioning process, and the distribution and scale characteristics of the artifacts in deep fake videos in an actual scene are simulated.
[0011] A local random elimination module is configured to shield local regions of a training sample under the guidance of the important region positioning module in the training process, and is essentially a data enhancement means. The input of the local random elimination module is the original training sample and the vertex coordinates of the regions output by the important region positioning module, and the output of the local random elimination module is a new training sample subjected to local elimination.
[0012] A multi-scale feature fusion module is configured to fuse features of different scales, and comprises a plurality of substructures in the form of "pooling-convolution-up sampling". The multi-scale feature fusion module extracts and fuses features of multiple scales in a face image into a new feature map. In an actual scene, a fake video contains small artifact regions, and the multi-scale feature fusion module can prevent these features from disappearing in the process of global average pooling by using feature maps of artifact regions of different scales. The input of the multi-scale feature fusion module is the feature map output by the important region positioning module, and the output of the multi-scale feature fusion module is a new feature map subjected to fusion.
[0013] The classifier module is used to give the classification result of the sample, which is composed of a global average pooling layer, a full connection layer and an activation function.
[0014] When the deep fake video detection system is in the training mode, the input is an enhanced image containing a face region, and the output is the normalized probability of the image belonging to two categories of true and false, and the normalized probability of each local region image belonging to two categories of true and false.
[0015] When the deep fake video detection system is in the inference mode, the input is an original image containing a face region, and the output is the normalized probability of the image belonging to two categories of true and false.
[0016] Further, the processing process of the data preprocessing module includes: extracting N frames at equal intervals from a video file, and extracting a face region in each frame using a trained MTCNN, and finally saving the image after face alignment, wherein N is determined according to the required data amount.
[0017] The application belongs to a deep fake video detection technology based on intra-frame images, so it is necessary to extract several independent frames from a video during training and inference. In order to reduce the interference of background information in the video, the application uses MTCNN to extract the face region in each frame. Considering that the resolution and video quality of the video file obtained in the actual situation often differ, in the data preprocessing module, the face region is randomly simulated at low resolution and added with noise. The specific operation is as follows: when simulating at low resolution, the original sample is down-sampled and then enlarged to the original size; when adding noise, Gaussian noise with a mean of 0 is added.
[0018] Further, the feature extraction module uses a modified ResNet-50 as a feature extraction network, and compared with the original ResNet-50 network structure, the step length of the convolution kernel in the ResBlock_4 layer is adjusted to 1, and the size of the feature map finally output by the layer is 14x14.
[0019] Further, the processing process of the important region positioning module includes: assuming that a feature map extracted from an image by ResNet-50 is F C×H×W , wherein C, H and W respectively represent the channel number, height and width of the feature map. As indicated in the following formula, by accumulating each channel of F, the corresponding activation map A can be obtained, wherein f i represents the component of the feature map F in the i-th channel, as indicated in the following formula:
[0020]
[0021] For activating a designated region r in the activation map, the activation degree of the region can be calculated by the following formula, where the coordinates of the upper left corner of the region are (x0, y0), and the coordinates of the lower right corner are (x1, y1).
[0022]
[0023] Then, the K most important regions are calculated using the Soft-NMS algorithm, as represented by the following formula, where A represents the activation map, t is the threshold used in the Soft-NMS algorithm, K is the number of selected regions, and represent the coordinates of the upper left corner and the lower right corner of the i-th region, respectively.
[0024]
[0025] The important region positioning module considers the sizes of eyes, nose, and mouth in a face relative to the overall face region, and uses sliding windows of various scales, with a minimum size of 3x3 and a maximum size of 7x10.
[0026] Further, the processing process of the local random elimination module includes: the present application selects windows of multiple sizes from the 14x14 feature map, wherein a smaller window (e.g., 3x3) can cover a complete facial feature, and a larger window (e.g., 7x10) can cover half a face. The present application randomly selects a region from the obtained K key regions, and when the area of the selected region is less than 1 / 3 of the original image, eliminates the pixel values corresponding to the region, and when the selected region is greater than 1 / 3 of the original image, only eliminates the pixel values of a random half of the region.
[0027] Further, the processing process of the multi-scale feature fusion module includes: first, a plurality of feature maps of corresponding scales are obtained using a plurality of pooling layers. Then, a 1x1 convolution kernel is used to reduce the dimension of the pooled feature map. Next, the small feature map of each scale is upsampled to the same size as the original feature map, and is spliced with the original feature map to form different channels of a new feature map. In actual scenarios, a fake video contains small fake regions, and the multi-scale feature fusion module can prevent these regions from disappearing in the global average pooling process. The present application uses two pooling layers with window sizes of 3 and 6.
[0028] Further, the feature extraction module and the multi-scale feature fusion module both use an SGD optimizer for network training optimization.
[0029] The application also provides a terminal device, comprising at least one processor, at least one memory, and computer program instructions stored in the memory, which realize the deepfake video detection system with dynamic composite feature enhancement when executed by the processor.
[0030] The application also provides a computer readable storage medium, which stores computer program instructions, which realize the deepfake video detection system with dynamic composite feature enhancement when executed by a processor.
[0031] Compared with the prior art, the application has the beneficial effects including:
[0032] The important region positioning module can dynamically extract and select the most discriminative region from the original training sample according to the feature map extracted by the CNN, and update the model parameters for these regions, so as to learn the artifact features of different regions. During training and inference, the key regions of the face do not need to be manually extracted, that is, other auxiliary models are not needed to segment the face region.
[0033] The local region elimination module uses the positioned important regions to guide and improve the random elimination strategy, so as to improve the generalization performance of the model, reduce the introduction of noise information, and prevent the convergence speed of the model from being affected.
[0034] The multi-scale feature fusion module differentiates the features extracted by the convolutional neural network into different scales and forms different channels of new feature maps, so as to prevent some subtle details from being discarded in the pooling process. The model can learn artifact features of different scales, so as to ensure high accuracy when encountering test samples of different resolutions and compression rates.
[0035] In summary, to solve the problem of insufficient generalization performance of the deepfake video detection algorithm, the application provides a deepfake video detection algorithm with dynamic composite feature enhancement, which introduces a multi-scale feature fusion module, an important region positioning module, and a local random elimination module, and improves the generalization ability of the model when encountering different compression rate datasets or unfamiliar datasets. BRIEF DESCRIPTION OF DRAWINGS
[0036] Figure 1 is a data flow diagram of the application in the training and testing stages;
[0037] Figure 2 is an operation diagram of the method proposed by the application in the preprocessing stage;
[0038] Figure 3are the parameters and output size of each layer in the network of the modified ResNet-50 used in the feature extraction stage of the present application;
[0039] Figure 4 is the flow chart in the training stage of the present application;
[0040] Figure 5 is the flow chart in the testing stage of the present application;
[0041] Figure 6 is the schematic diagram of the principle of selecting local key regions according to feature maps in the training stage of the present application;
[0042] Figure 7 is the schematic diagram of the principle of randomly selecting occluded regions according to the selected local key regions in the training stage of the present application;
[0043] Figure 8 is the network structure schematic diagram of the multi-scale feature fusion module used in the training stage of the present application. DETAILED DESCRIPTION
[0044] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings.
[0045] The embodiments of the present application and the implementation process thereof are as follows:
[0046] Please refer to Figures 1-8 A deep fake video detection system, comprising a data preprocessing module, a feature extraction module, an important region positioning module, a local random elimination module, a multi-scale feature fusion module and a classifier module.
[0047] The data preprocessing module is used to extract the face region from the video, as shown in Figure 2 The present application belongs to a deep fake video detection technology based on intra-frame images, and therefore needs to extract several independent frames from the video during training and reasoning. In order to reduce the interference of background information in the video, the present application uses MTCNN to extract the face region in each frame. After obtaining the face region, two data enhancement techniques of "random horizontal flip" and "random angle rotation" are used. The input is a video file, and the output is multiple enhanced images containing face regions. Among them, the probability of using "random horizontal flip" enhancement operation is 50%, the probability of using "random angle rotation" is 50%, and the rotation angle is a random value in [-10°, 10°].
[0048] The feature extraction module is used to extract features from the global or local region of the face. The input is a complete face image or a local region of the face image, and the output is a feature map with a size of 14x14.
[0049] The important region positioning module is used to extract the most discriminative regions from the samples in the training process, i.e., to locate the most important regions, and contains a non-maximum suppression (NMS) screening layer. The input is a 14x14 feature map extracted from a face image by the feature extraction module, and the output is the vertex coordinates of the extracted multiple regions and the feature map.
[0050] The local random elimination module is used to mask the local regions of the training samples under the guidance of the important region positioning module in the training process. The input is the original training sample and the vertex coordinates of the multiple regions output by the important region positioning module, and the output is a new training sample after local elimination.
[0051] The multi-scale feature fusion module is used to fuse features of different scales, and contains multiple sub-structures of the form of "pooling-convolution-up sampling". The input is the feature map output by the important region positioning module, and the output is a new feature map after fusion.
[0052] The classifier module is used to give the classification result of the sample, which consists of a global average pooling layer, a fully connected layer and a Sigmoid activation function. The input is the feature map output by the multi-scale feature fusion module, and the output is the final classification result.
[0053] When the deep fake video detection system is in the training mode, the input is an enhanced image containing a face region, and the output is the normalized probability of the image belonging to the true or fake class, and the normalized probability of each local region image belonging to the true or fake class.
[0054] When the deep fake video detection system is in the inference mode, the input is an original image containing a face region, and the output is the normalized probability of the image belonging to the true or fake class.
[0055] In this embodiment, the feature extraction network uses a modified ResNet-50 as the backbone network of the feature extraction module, and its structure is as shown in Figure 3 Compared with the original ResNet-50, the step of the convolution kernel in the ResBlock_4 layer is adjusted to 1, and the size of the feature map finally output by this layer is 14x14.
[0056] Further, the processing process of the important region positioning module includes: as shown in Figure 4 , it is assumed that the feature map extracted from an image by ResNet-50 is F e R C×H×W , C, H, and W represent the channel number, height, and width of the feature map, respectively. As shown in the following formula, the activation map A corresponding to F can be obtained by accumulating each channel of F, where f i represents the component of the feature map F in the i-th channel.
[0057]
[0058] For activating a specified region r in the activation map, the activation degree of the region can be calculated by the following formula, where the coordinates of the upper left corner of the region are (x0, y0), and the coordinates of the lower right corner are (x1, y1).
[0059]
[0060] Then, as shown in Figure 6 , the Soft-NMS algorithm is used to calculate the K most important regions. Where A represents the activation map, t is the threshold used in the Soft-NMS algorithm, which is selected as 0.3 in this embodiment; K is the number of selected regions, and respectively represent the coordinates of the upper left corner and the lower right corner of the i-th region, which are logically represented by the following formula.
[0061]
[0062] In this embodiment, a plurality of window sizes of local regions are selected, wherein the small window selects 4x4, 3x5, and 5x3 shapes, the medium window selects 6x6, 5x7, and 7x5, and the large window selects 8x8, 7x10, and 10x7. For each training sample, two large, medium, and small windows are selected in this embodiment, which can reduce the value of K when the video memory is limited, and can appropriately increase the value of K when the video memory resource is relatively abundant.
[0063] The processing process of the local random elimination module includes: as shown in Figure 7 , the present application selects a plurality of size windows from the 14x14 feature map, and randomly selects a region from the K key regions obtained by the present application. When the selected region is the above-mentioned small and medium window, the pixel value corresponding to the region is eliminated, and when the selected region is the above-mentioned large window, only the pixel value of a random half region in the region is eliminated.
[0064] The processing process of the multi-scale feature fusion module includes: as shown in Figure 8 , first, a plurality of feature maps corresponding to different scales are obtained by using a plurality of pooling layers. Then, a 1x1 convolution kernel is used to reduce the dimension of the pooled feature map. Next, the small feature map of each scale is up-sampled to the same size as the original feature map, and is spliced into different channels of the new feature map. In this embodiment, two pooling layers with window sizes of 3 and 6 are used.
[0065] In this embodiment, the Focal loss with an added alpha-balance factor is used as the loss function of the classification of the global region and the local region, which increases the weight of complex samples in the overall loss.
[0066] L global = loss(P(X g ), Y)
[0067]
[0068] where X g is the input original image, is the i-th local region, K is the number of extracted local regions, Y is the true class of the sample, and P is the probability output by the SoftMax layer of the model. The final loss function is obtained by weighted summation of the two parts of the loss:
[0069] Loss = L global + λL parts
[0070] λ is set to 0.5 in the first epoch and 1 in the subsequent epochs.
[0071] In the embodiment, the feature extraction module and the multi-scale feature fusion module provide the SGD optimizer with the following specific coefficient selection:
[0072] The learning rate of the feature extraction module is initialized to 1e-5, and the learning rates of the multi-scale feature fusion module and the classifier module are initialized to 1e-3. The learning rate of all parameters is adjusted to 0.2 of the original value every 5 epochs, and the training is performed for a total of 20 epochs.
[0073] In the embodiment, when in the test mode, as shown in Figure 5 , the important region positioning module and the local random elimination module are not used in the process of forward transmission, that is, the dynamic data augmentation technology applied in the training mode is not applied to the sample. After the input face test sample passes through the feature extraction module and the multi-scale feature fusion module, the classifier module gives the class to which the sample belongs.
[0074] The embodiment of the application further discloses a terminal device.
[0075] A terminal device includes a processor, a memory and a network interface connected through a system bus. Wherein, the processor of the device is used to provide computing and control ability. The memory of the device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the device is used to store service requests, service data and other data. The network interface of the device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a data processing method.
[0076] The embodiment of the present application further discloses a computer readable storage medium.
[0077] A storage medium stores a computer program, which is executed by a processor to implement the steps in the above system embodiments.
[0078] A person of ordinary skill in the art can understand that all or part of the processes in the above system embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the computer program can include the processes of the above system embodiments. Any reference to memory, storage, database or other medium in the embodiments of the present application can include non-volatile and / or volatile memory. The non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. The volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0079] The above embodiments are only used to illustrate the technical solutions of the present application but not to limit the present application. Although the present application is described in detail with reference to the above embodiments, a person of ordinary skill in the art should understand that the specific embodiments of the present application can be modified or replaced equivalently without departing from the spirit and scope of the present application, and any modification or equivalent replacement without departing from the spirit and scope of the present application should be covered in the scope of the claims of the present application.
Claims
1. A deepfake video detection system, comprising: Comprise: Data preprocessing module for extracting face region from video, its input video file, output multiple enhanced containing face region image, enhancement strategy for adding Gaussian noise and actual scene in different video quality compression changes simulation; Feature extraction module for extracting features from the global or local region of the face, its input is the complete face image or local area, output the feature map extracted by deep neural network; Important area positioning module for extracting the most discriminative region from the sample in the training process, its input is the feature map extracted by the feature extraction module, and the output is the vertex coordinates and feature map of the extracted multiple regions. In the positioning process, different size sliding windows are selected, and the distribution and scale characteristics of the artifacts in the deep fake video in the actual scene are simulated; Local random elimination module for occluding the local area of the training sample under the guidance of the important area positioning module, its input is the original training sample and the vertex coordinates of the multiple regions output by the important area positioning module, and the output is the new training sample after local elimination; Multi-scale feature fusion module for fusing features of different scales, containing multiple "pooling-convolution-up sampling" substructures, its input is the feature map output by the important area positioning module, and the output is the new feature map after fusion; And Classifier module for giving the classification result of the sample, composed of global average pooling layer, fully connected layer and activation function, its input is the feature map output by the multi-scale feature fusion module, and the output is the final classification result.
2. The deepfake video detection system of claim 1, wherein, The processing process of the data preprocessing module includes: extracting N frames from the video file at equal intervals, and extracting the face region in each frame using the trained MTCNN. The finally saved image is aligned with the face, wherein N is determined according to the required data amount.
3. The deepfake video detection system of claim 1, wherein, The feature extraction module uses a modified ResNet-50 as the feature extraction network. Compared with the original ResNet-50 network structure, the step length of the convolution kernel in the ResBlock_4 layer is adjusted to 1, and the size of the feature map finally output by this layer is 14x14.
4. The deepfake video detection system of claim 1, wherein, The processing procedure of the important region positioning module includes: a feature map F extracted from an image by ResNet-50, F element of R C×H×W , wherein C, H and W respectively represent the channel number, height and width of the feature map, and an activation map A is obtained by accumulating each channel of F, wherein f i represents a component of the feature map F in the i-th channel, as shown in the following formula: For a specified region r in the activation map, the activation degree of the region can be calculated using the following formula, and the coordinates of the top left corner of the region are (x0, y0), and the coordinates of the bottom right corner are (x1, y1); Then, the K most important regions are calculated using the Soft-NMS algorithm, where A represents the activation map, t is the threshold used in the Soft-NMS algorithm, and K is the number of regions selected, and respectively represent the coordinates of the upper left corner and the lower right corner of the i-th region, 5. The deepfake video detection system of claim 1, wherein, The processing process of the local random elimination module includes: selecting multiple size windows from the feature map, randomly selecting a block from the K key regions obtained, and eliminating the pixel values corresponding to the region.
6. The deepfake video detection system of claim 1, wherein, The processing process of the multi-scale feature fusion module includes: obtaining multiple feature maps containing small artifact regions of corresponding scales by using multiple different pooling layers, then using a 1x1 convolution kernel to reduce the dimension of the pooled feature map, then up-sampling each small feature map of each scale to the same size as the original feature map, and concatenating the original feature map into different channels of the new feature map.
7. A terminal device, characterized by comprising: The terminal device comprises: At least one processor, at least one memory, and computer program instructions stored in the memory that, when executed by the processor, implement the deepfake video detection system of any of claims 1-6.
8. A computer-readable storage medium, characterized in that, A computer storage medium having stored thereon computer program instructions that, when executed by a processor, implement the deepfake video detection system of any of claims 1-6.