A Deepfake video tampering detection method and system

Abstract

The present invention discloses a Deepfake video tampering detection method and system. The present invention is based on the idea of ​​subspace continuous learning. Different subspace areas of a frame image sequence in a target video are all subjected to feature extraction by a cascade network. The cascade network learns features of different subspace areas of the same frame image respectively, and can effectively utilize the unbalanced traces left by tampering. In addition, while learning pixel neighborhood features, the cascade network also takes into account the correlation features of three channels of color images, and fully captures the feature relationship between facial subspace areas and the color distribution features between RGB image channels, which can improve the accuracy of tampering detection of the target video. Moreover, compared with the existing deep learning network based on data-driven CNN or GAN, the cascade network used in the present invention has fewer parameters, greatly reduces the time of feature learning, and improves the overall tampering detection efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of video tampering detection, and in particular to a Deepfake video tampering detection method and system. Background Art

[0002] Deepfake video tampering is a hot research topic in the field of computer vision and image-based forensics. It has high practical value in many aspects such as face review, news authentication, portrait rights protection, pornographic image review, etc., and protects the information security of the general public. The main task of Deepfake video tampering detection is to be able to identify the authenticity of a large number of video media circulating on the Internet, and to determine whether the identity information and facial expressions of a certain person in the video have been artificially modified. Compared with the currently popular Deepfake video tampering detection based on deep learning methods and models, non-deep learning identification models can show the advantages of saving more time and reducing the use of computer resources.

[0003] The face tampering detection model based on deep learning has a great dependence on the data set. When the performance is tested on different data sets, their identification accuracy will decrease. This is mainly because the tampering detection method based on deep learning needs to learn the new features generated by a type of tampering technology in order to effectively identify the traces left by the tampering technology. Therefore, whenever a new tampering technology appears, it will always pose a challenge to the existing detection methods. The identification model based on non-deep learning can learn the classification features precisely and directionaly for the underlying features of the image when learning the features of the image, and it is also more interpretable.

[0004] The existing Deepfake video tampering detection solutions still have the following defects:

[0005] 1) Existing detection models generally use data-driven CNN or GAN deep learning networks, which generally have hundreds of thousands or millions of parameters, resulting in a large amount of overall computation;

[0006] 2) Existing detection models usually use single-channel pixel images as input, which will lose the color distribution characteristics between RGB image channels and fail to effectively utilize the color distribution imbalance traces left by video tampering. Summary of the invention

[0007] The present invention aims to at least solve the technical problems existing in the prior art. To this end, the present invention proposes a Deepfake video tampering detection method and system, which can improve the accuracy of video tampering detection.

[0008] A first aspect of the present invention provides a Deepfake video tampering detection method, comprising the following steps:

[0009] Filtering a frame image sequence from a target video, wherein each frame image in the frame image sequence contains a human face region;

[0010] Dividing the facial region of a human face in each frame image into a plurality of sub-frame images, each of the sub-frame images representing one of the subspace regions divided from the facial region of the human face;

[0011] Performing feature extraction on all the sub-frame images through a preset cascade network; wherein, performing feature extraction on the sub-frame images through the cascade network comprises:

[0012] Convolution is performed on the subframe image, pixel neighborhood correlation feature extraction is performed on the result after convolution, the feature extraction result is pooled, and the pooling result is characterized to obtain first feature map components of high frequency, medium frequency and low frequency; convolution is performed on the first feature map component of low frequency, pixel neighborhood correlation feature extraction is performed on the result after convolution, the feature extraction result is pooled, and the pooling result is characterized to obtain second feature map components of high frequency, medium frequency and low frequency; and so on, until convolution is performed on the N-1th feature map component of low frequency, pixel neighborhood correlation feature extraction is performed on the result after convolution, the feature extraction result is pooled, and the pooling result is characterized to obtain Nth feature map component of high frequency, medium frequency and low frequency; wherein N refers to the maximum number of layers of the cascade network;

[0013] Generate a feature vector according to the first feature map component to the Nth feature map component of all intermediate frequencies;

[0014] Video tampering detection is performed according to the feature vector.

[0015] According to the embodiments of the present invention, there are at least the following technical effects:

[0016] The method firstly selects frame images containing human face areas from the target video; then the human face areas in each frame image are divided non-repetitively into multiple subspace areas to obtain multiple subframe images; then the multiple subframe images are used as input items of the cascade network, and the features of the multiple subspace areas of each input frame image are extracted through each layer of subnetwork in the cascade network, wherein the first layer of subnetwork of the cascade network performs the first convolution, pixel neighborhood correlation feature extraction, pooling processing and feature representation on the input subspace area to obtain high-frequency, low-frequency and medium-frequency feature map components, discards the high-frequency feature map components, and uses the low-frequency feature map components as the input of the next layer of subnetwork, and so on, until the last layer of subnetwork outputs the high-frequency, low-frequency and medium-frequency feature map components; then the low-frequency feature map components generated by each layer of subnetwork generate feature vectors, and finally prediction is performed based on the feature vectors to detect whether the target video is tampered with.

[0017] This method is based on the idea of ​​subspace continuous learning. Different subspace areas from the frame image sequence in the target video are all subjected to feature extraction by the cascade network. The cascade network learns features for different subspace areas of the same frame image respectively, which can effectively utilize the unbalanced traces left by tampering. In addition, while learning pixel neighborhood features, the cascade network also takes into account the correlation features of the three channels of the color image, which fully captures the feature relationship between the facial subspace areas and the color distribution features between the RGB image channels, and can improve the accuracy of tampering detection of the target video. Moreover, compared with the existing deep learning network based on data-driven CNN or GAN, the cascade network used in this method has fewer parameters, which greatly reduces the time of feature learning and improves the overall tampering detection efficiency.

[0018] According to some embodiments of the present invention, the step of filtering out a frame image sequence from a target video comprises the following steps:

[0019] Selecting all frame images containing human face areas from the target video in sequence;

[0020] Calculating the pixel matrix distance between two adjacent frame images, when the pixel matrix distance between two adjacent frame images is greater than a threshold, retaining the two adjacent frame images; when the pixel matrix distance between two adjacent frame images is less than or equal to the threshold, deleting the frame image with lower definition among the two adjacent frame images;

[0021] The retained frame images are combined into a frame image sequence.

[0022] According to some embodiments of the present invention, the step of calculating the pixel matrix distance between two adjacent frame images further includes the steps of:

[0023] The color frame image is converted into a grayscale frame image, and the pixel matrix distance between two adjacent grayscale frame images is calculated.

[0024] According to some embodiments of the present invention, before dividing the facial area of ​​each frame image into a plurality of sub-frame images, the step of:

[0025] When the size of the face area of ​​the frame image is smaller than a preset value, the frame image smaller than the preset value is super-resolutioned through the RCAN network.

[0026] According to some embodiments of the present invention, the step of generating a feature vector according to the first feature map component to the Nth feature map component of all intermediate frequencies comprises the steps of:

[0027] The PCA is used to perform principal component analysis on the first to Nth feature map components of the intermediate frequency and retain the set energy value to generate a feature vector.

[0028] According to some embodiments of the present invention, after generating the feature vector, the method further includes:

[0029] The feature vector is assigned a classification label through the XGBoost classifier.

[0030] According to some embodiments of the present invention, the cascade network consists of 4 layers of sub-networks, the convolution kernel size of the first two layers of sub-networks is 2*2, and the step size is 1; the convolution kernel size of the last two layers of sub-networks is 3*3, and the step size is 1.

[0031] A second aspect of the present invention provides a Deepfake video tampering detection system, comprising:

[0032] A frame image screening module, used for screening out a frame image sequence from a target video, wherein each frame image in the frame image sequence contains a human face area;

[0033] A subspace division module, used for dividing the face area of ​​each frame image into a plurality of subframe images, each of which represents one of the subspace areas divided from the face area;

[0034] A cascade network module is used to extract features from all the sub-frame images, wherein extracting features from the sub-frame images through the cascade network includes:

[0035] Convolution is performed on the subframe image, pixel neighborhood correlation feature extraction is performed on the result after convolution, the feature extraction result is pooled, and the pooling result is characterized to obtain first feature map components of high frequency, medium frequency and low frequency; convolution is performed on the first feature map component of low frequency, pixel neighborhood correlation feature extraction is performed on the result after convolution, the feature extraction result is pooled, and the pooling result is characterized to obtain second feature map components of high frequency, medium frequency and low frequency; and so on, until convolution is performed on the N-1th feature map component of low frequency, pixel neighborhood correlation feature extraction is performed on the result after convolution, the feature extraction result is pooled, and the pooling result is characterized to obtain Nth feature map component of high frequency, medium frequency and low frequency; wherein N refers to the maximum number of layers of the cascade network;

[0036] A feature extraction module, used for generating a feature vector according to the first feature map component to the Nth feature map component of all intermediate frequencies;

[0037] A decision module is used to perform video tampering detection according to the feature vector.

[0038] According to the embodiments of the present invention, there are at least the following technical effects:

[0039] The system first uses a frame image screening module to filter out frame images containing facial areas from the target video; then uses a subspace division module to divide the facial areas in each frame image without duplication, divides multiple subspace areas, and obtains multiple subframe images; then uses a cascade network module to use multiple subframe images as input items of the cascade network, and extracts features from multiple subspace areas of each input frame image through each layer of subnetwork in the cascade network, wherein the first layer of the cascade network performs convolution, pixel neighborhood correlation feature extraction, pooling processing and feature representation on the input subspace area to obtain high-frequency, low-frequency and medium-frequency feature map components, discards the high-frequency feature map components, and uses the low-frequency feature map components as the input of the next layer of subnetwork, and so on, until the last layer of subnetwork outputs high-frequency, low-frequency and medium-frequency feature maps; then uses the feature extraction module to generate feature vectors from the medium-frequency feature map components generated by each layer of subnetwork, and finally uses the decision module to predict based on the feature vectors to detect whether the target video has been tampered with.

[0040] This system is based on the idea of ​​subspace continuous learning. Different subspace areas from the frame image sequence in the target video are all subjected to feature extraction by the cascade network. The cascade network learns features for different subspace areas of the same frame image respectively, which can effectively utilize the unbalanced traces left by tampering. In addition, while learning pixel neighborhood features, the cascade network also takes into account the correlation features of the three channels of color images. It fully captures the feature relationship between facial subspace areas and the color distribution features between RGB image channels, which can improve the accuracy of tampering detection of the target video. Moreover, compared with the existing deep learning networks based on data-driven CNN or GAN, the cascade network used in this system has fewer parameters, which greatly reduces the time of feature learning and improves the overall tampering detection efficiency.

[0041] According to a third aspect of the present invention, a Deepfake video tampering detection device is provided, comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the Deepfake video tampering detection method as described above.

[0042] A fourth aspect of the present invention is a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the Deepfake video tampering detection method as described above.

[0043] It should be noted that the beneficial effects between the third aspect to the fourth aspect of the present invention and the prior art are the same as the beneficial effects between the above-mentioned Deepfake video tampering detection method and the prior art, and will not be described in detail here.

[0044] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0046] Figure 1 It is a flowchart of a Deepfake video tampering detection method provided by an embodiment of the present invention;

[0047] Figure 2 is a flowchart of a Deepfake video tampering detection method provided by another embodiment of the present invention;

[0048] Figure 3 It is a flowchart of a Deepfake video tampering detection method provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0049] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0050] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "axial", "radial", "circumferential" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0051] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0052] First embodiment

[0053] Deepfake video tampering is a hot research topic in the field of computer vision and image forensics. It has high practical value in many aspects such as face review, news authenticity, portrait rights protection, pornographic image review, etc., and protects the information security of the general public. The main task of Deepfake video tampering detection is to be able to identify the authenticity of the characters in a large number of video media circulating on the Internet, and to determine whether the identity information and facial expressions of a certain person in the video have been artificially modified.

[0054] The existing Deepfake video tampering detection solutions still have the following defects:

[0055] 1) Existing detection models generally use data-driven CNN or GAN deep learning networks, which generally have hundreds of thousands or millions of parameters, resulting in a large amount of overall computation;

[0056] 2) Existing detection models usually use single-channel pixel images as input, which will lose the color distribution characteristics between RGB image channels and fail to effectively utilize the color distribution imbalance traces left by video tampering.

[0057] In order to solve the above technical defects, refer to Figure 1 An embodiment of the present invention provides a Deepfake video tampering detection method, comprising the following steps:

[0058] Step S101: Filter out a frame image sequence from a target video, wherein each frame image in the frame image sequence contains a human face region.

[0059] Among them, the target video refers to the video that needs to be judged whether face tampering occurs (also called the video that needs to be judged whether the authenticity of the face is genuine); the frame image sequence is composed of multiple frame images; in some embodiments, frame images containing facial areas of the face can be extracted from the target video through OpenCV or Matlab, which is common knowledge of those skilled in the art and will not be described in detail in this embodiment.

[0060] Step S103, dividing the facial region of the human face in each frame image into a plurality of sub-frame images, each sub-frame image representing one of the subspace regions divided from the facial region of the human face. In some embodiments, the facial region of the human face in the frame image is divided into five subspace regions based on the five views of the human face, that is, five sub-frame images are obtained. It should be noted that the sub-frame image is used here to represent the sub-image divided from the frame image.

[0061] Step S105: extracting features from all sub-frame images through a preset cascade network; wherein extracting features from sub-frame images through a cascade network includes:

[0062] Convolve the subframe image, extract pixel neighborhood correlation features from the convolution result, pool the feature extraction result, and perform feature representation on the pooling result to obtain high-frequency, medium-frequency and low-frequency first feature map components;

[0063] Convolve the low-frequency first feature map component, extract pixel neighborhood correlation features from the convolution result, pool the feature extraction result, and perform feature representation on the pooling result to obtain high-frequency, medium-frequency and low-frequency second feature map components;

[0064] And so on, until the N-1th feature map component of the low frequency is convolved, the pixel neighborhood correlation feature is extracted from the convolution result, the feature extraction result is pooled, and the pooling result is represented by features to obtain the high frequency, medium frequency and low frequency Nth feature map components; where N refers to the maximum number of layers of the cascade network.

[0065] In step S105, the cascade network is based on the idea of ​​Successive Subspace Learning (SSL), and each layer of sub-network in the cascade network extracts features from multiple sub-space regions of each input frame image. The first layer of sub-network performs convolution, pixel neighborhood correlation feature extraction, pooling, and feature representation on the input sub-space region to obtain high-frequency, low-frequency, and medium-frequency feature map components, and uses the low-frequency feature map components as the input of the next layer of sub-network, and so on, until the last layer of sub-network outputs high-frequency, low-frequency, and medium-frequency feature map components (from top to bottom, the local features of the feature map output by each layer of sub-network gradually decrease, and the global features gradually increase; from left to right, the feature dimension decreases, and the feature expression ability increases). The feature map components output by the first layer of sub-network have more local feature details, while the feature map components output by the last layer have more global feature description capabilities.

[0066] Step S107: Generate a feature vector according to the first feature map component to the Nth feature map component of all intermediate frequencies.

[0067] Step S109: Perform video tampering detection based on the feature vector.

[0068] This method is based on the idea of ​​subspace continuous learning. Different subspace areas from the frame image sequence in the target video are all subjected to feature extraction by the cascade network. The cascade network learns features for different subspace areas of the same frame image respectively, which can effectively utilize the unbalanced traces left by tampering. In addition, while learning pixel neighborhood features, the cascade network also takes into account the correlation features of the three channels of the color image, which fully captures the feature relationship between the facial subspace areas and the color distribution features between the RGB image channels, and can improve the accuracy of tampering detection of the target video. Moreover, compared with the existing deep learning network based on data-driven CNN or GAN, the cascade network used in this method has fewer parameters, which greatly reduces the time of feature learning and improves the overall tampering detection efficiency.

[0069] Second embodiment

[0070] Reference Figure 2 Based on the above embodiment, in step S101, a frame image sequence is screened out from a target video, including the steps of:

[0071] Step S1011, all frames containing human face areas are selected from the target video in sequence. For the target video, step S1011 detects color frames containing human faces by traversing frame by frame.

[0072] Step S1012, converting the color frame image into a grayscale frame image, and using the grayscale frame image to calculate the pixel matrix distance. This processing can improve the calculation speed of the pixel matrix distance between two subsequent adjacent frame images. It should be noted that in this article, the color frame image is screened from the target video. The color frame image is converted into a grayscale frame image to improve the calculation efficiency of calculating the pixel matrix distance. The frame images processed subsequently are all color frame images.

[0073] Step S1013, calculate the pixel matrix distance between two adjacent frame images, when the pixel matrix distance between two adjacent frame images is greater than the threshold, retain the two adjacent color frame images; when the pixel matrix distance between two adjacent frame images is less than or equal to the threshold, delete the color frame image with lower clarity in the two adjacent frame images. Step S1013, based on all the frame images selected in step S1011, retains frame images with lower similarity and higher clarity, removes low-quality frame images in the video, and only retains frame images with clear faces and low repetitiveness, which improves the efficiency and resource occupancy of the cascade network in learning features and reduces data redundancy for subsequent network training and testing.

[0074] Step S1014: Group the retained frame images into a frame image sequence.

[0075] Before step S103, the method further includes the following steps:

[0076] Step S102, when the size of the face area of ​​the frame image is smaller than the preset value, the frame image smaller than the preset value is super-resolved through the RCAN network. So that the size of the face area of ​​the frame image after super-resolution processing meets the preset value. The main function of step S102 is to control the size of the face area in the frame image. When the size of the face area of ​​the frame image is smaller than the preset value, the frame image smaller than the preset value is super-resolved through the RCAN network. While changing the size of the frame image, it also enhances the high-frequency information that can be used in subsequent feature extraction (that is, it enhances the high-frequency feature information of the color distribution between the RGB image channels to improve the processing effect of the subsequent cascade network). It should be noted that there is no specific restriction on the preset value here, and it can be set according to actual needs.

[0077] Based on the above embodiment, the cascade network consists of 4 layers of sub-networks, the convolution kernel size of the first two layers of sub-networks is 2*2, and the step size is 1; the convolution kernel size of the last two layers of sub-networks is 3*3, and the step size is 1.

[0078] Based on the above embodiment, in step S107, all low-frequency first feature map components to N-th feature map components are used to generate feature vectors, including: performing principal component analysis on the low-frequency first feature map components to N-th feature map components through PCA and retaining the set energy value to generate feature vectors. Performing secondary feature extraction on the low-frequency first feature channel to N-th feature channel generated in step S105 through PCA (Principal Components Analysis) can achieve the purpose of enhancing feature expression and reducing dimensionality.

[0079] Third embodiment

[0080] An embodiment of the present invention provides a Deepfake video tampering detection system, which includes: a frame image screening module, an RCAN network module, a subspace partitioning module, a cascade network module, a feature extraction module, an aggregation module, and a decision module. The system performs a Deepfake video tampering detection method, which includes:

[0081] Step S201: sampling frame images from a target video through a frame image screening module.

[0082] For the input target video, the color image is first converted into a grayscale image to improve the speed of subsequent pixel matrix distance calculation. The first frame image containing the face area in the target video is cached as a reference, and its facial pixel matrix features are recorded after alignment processing, and its corresponding color image is saved.

[0083] Each frame image containing the facial area of ​​a human face will be compared with the previous reference frame image, and the pixel matrix distance between the two will be compared. When the distance is greater than the threshold value of 0.3, it is considered that the two have a large change in facial features, and both are saved. The image distance calculation formula is:

[0084]

[0085] in, Represents the i-th pixel value of the pixel matrix of the previous grayscale frame image, is the i-th pixel value of the pixel matrix of the next grayscale frame image. After experimental analysis, The threshold value is preferably 0.3.

[0086] If the pixel matrix distance between the two frame images is less than or equal to the threshold value of 0.3, the two color frame images are considered similar images. At this time, the Laplace algorithm is used to calculate and compare the clarity of the two frame images, and the clearer frame image is retained. The clarity comparison expression is:

[0087]

[0088] in, It is the clearer image between image A and image B. and They represent the clarity index values ​​of image A and image B respectively. These values ​​are obtained by the Laplacian algorithm in OpenCV and are used to analyze the clarity of the image.

[0089] Step S202: super-resolution processing is performed on the frame image whose face area size is too small through the RCAN network module.

[0090] For all retained frames, check the size of the facial area, and retain the appropriate size directly. If the size of the facial area is too small (for example, less than 68*128), use the RCAN network to perform super-resolution processing on it. While achieving the purpose of changing the image size, it also enhances the high-frequency feature information of the color distribution between the RGB image channels, so as to facilitate the related processing of the subsequent cascade network module. Frame images with relatively small facial areas will be super-resolved through the RCAN network.

[0091] Step S203: non-repetitively divide the facial area of ​​the human face through the subspace division module. The same frame image is divided into subframe images of different regions, and the divided images are used as input data sets of the cascade network module.

[0092] Step S204: performing a first feature extraction on the subframe image through a cascade network module.

[0093] For the input single subspace region image, the first feature extraction is performed first, and several feature maps will be obtained. The feature maps are divided into three categories: high frequency, medium frequency, and low frequency. The high-frequency feature map components obtained by the first layer of sub-network are discarded, and the medium-frequency feature map components enter the PCA principal component analysis for dimensionality reduction and feature distillation. The low-frequency feature map components are used as the input of the second layer of sub-network. Similarly, high-frequency, medium-frequency, and low-frequency feature map components are obtained. And so on, until the feature extraction of four layers of sub-networks. When the frame sequence from the same video and the different subspace regions from the frame sequence have all been extracted by the cascade network, the feature map of each layer has gradually reduced detailed features and gradually enhanced global features from top to bottom; from left to right, the feature dimension decreases and the feature expression ability increases.

[0094] Step S205: Perform principal component analysis on all low-frequency feature maps through a feature extraction module and set different pre-classification labels.

[0095] PCA is used for deep extraction (to enhance the expressiveness of features) and dimensionality reduction. The feature graph generated by the cascade network will be subjected to secondary dimensionality reduction and feature extraction after entering the PCA principal component analysis. As a result, the dimension of the feature graph is greatly reduced, and the corresponding feature vector channel is obtained. Each feature graph corresponds to a feature vector channel. The feature vectors in the training phase will be classified into pre-classification labels according to the source of the data set. When setting the pre-classification label, the XGBoost classifier is used. The parameter scale of the XGBoost classifier is suitable for this module. The pre-classification label is set for each feature vector by training and testing the XGBoost classifier.

[0096] Step S206: Aggregate different pre-classified labels, target video, sampled frame images, and subspace region information through an aggregation module. The aggregation module is equivalent to a container for storing label information, which is used to aggregate different subspace regions of sampled frame images on the one hand, and to aggregate all sampled frame images of the target video on the other hand.

[0097] Step S207: Perform detection through a decision module.

[0098] The decision module uses an average weighted approach based on the number of pre-classified labels of two types (true and false) and makes a decision on the possibility that the video is true based on the confidence threshold range, and will obtain the detection result of video tampering.

[0099] like Figure 3 As shown, the specific process of Deepfake video tampering detection is given;

[0100] The subspace regions of size 68*68*3 are numbered to correspond to the position of a subspace region in a certain frame of the video, where 68*68 is the image size and 3 represents the three RGB channels of the image.

[0101] A 68*68*3 image first enters the first layer of the cascade network ( Figure 3 1st-PixelHop++ is used in the representation). The convolution kernel size of this layer is 2*2 and the step size is 1. After being processed by the two modules Neighborhood Construction (for pixel neighborhood correlation feature extraction) and C / W Saab transform (for feature representation), it is then pooled by (2*2)-(1*1) ( Figure 3Max-Pooling is used in the representation). Among them, Neighborhood Construction is similar to the convolution kernel function of the CNN network. Compared with the feature learning of the grayscale image, this embodiment uses Neighborhood Construction as the "convolution kernel" to capture the characteristics of the color image, so as to achieve the purpose of extracting the pixel neighborhood features of the color image.

[0102] The convolution kernel of the second layer is 2*2, and the convolution kernel of the third and fourth layers is 3*3, and the step size is 1.

[0103] After the first layer of cascade network, we get K1=K 11 +K 12 +K 13 The number of feature maps, where:

[0104] K 13 Feature channels representing high frequencies and poor correlation are discarded;

[0105] K 11 The feature channels representing medium-frequency and weakly correlated features directly enter PCA ( Figure 3 PCA is used in the representation);

[0106] K 12 The feature channels representing low frequency and strong correlation are sent to the second layer ( Figure 3 It is important to note that the cascade network will automatically divide the feature channels into high-frequency, medium-frequency and low-frequency channels. High frequency refers to the fast change of image details (edges and contours), low frequency refers to the opposite, and medium frequency is between the two.

[0107] After cascading features layer by layer in this way, the feature map of the first layer has more local feature details, while the feature map of the fourth layer has better description capabilities of global features. While learning pixel neighborhood features, the cascade network also takes into account the correlation features of the three channels of color images.

[0108] Then for the number K 11 , K 21 , K 31 , K 41 The feature vector channel enters the second feature extraction stage, that is, enters the feature extraction module ( Figure 3 After using Feature Distillation Module in ), PCA will perform principal component analysis on each feature channel and retain 90% of the energy, save it as a feature vector, and bind the pre-classification label ( Figure 3Pre-Classifiers are used in the representation, and Pre-Classifiers use XGBoost classifier).

[0109] Aggregation Module ( Figure 3 Feature Concatenation Module is used in the feature concatenation module to aggregate the labels generated by Pre-Classifiers and the corresponding videos, sampled frame images, and subspace region information. The aggregation module contains two parts: one is the aggregation of different subspace regions of the frame image, and the other is the aggregation of all frame images of the same video.

[0110] Finally, for a video, the decision module ( Figure 3 Using the Decision module) there will be E=(K 11 +K 21 +K 31 +K 41 )*n*N pre-classified label vectors, where n=5 represents the number of pre-fetched subspaces, and N represents the number of frames with facial regions retained in a video, which is usually dozens. The decision module uses an average weighted approach to calculate the authenticity of the detected video for the E pre-classified label vectors from the same video.

[0111] Compared with the existing Deepfake video detection solutions, the Deepfake video tampering detection system and method provided in this embodiment have the following advantages:

[0112] (1) Compared with the detection models based on CNN and GAN, this system first removes low-quality frame images in the video and only retains frame images with clear faces and low repetitiveness, thereby improving the efficiency and resource utilization of the system when learning features.

[0113] (2) In the field of Deepfake video tampering, the image size is processed using RCAN super-resolution processing. While changing the image size, it also enhances the high-frequency information that can be used in feature extraction (i.e., it enhances the high-frequency feature information of color distribution between RGB image channels to improve the subsequent cascade network processing effect);

[0114] (3) This system is based on the SSL concept, and all the different subspace regions of the frame image from the target video are subjected to feature extraction by the cascade network. The cascade network learns features from different subspace regions of the same frame image separately, and can make more effective use of the unbalanced traces left by tampering technology; in addition, while learning pixel neighborhood features, the cascade network also takes into account the correlation features of the three channels of the color image, fully capturing the feature relationship between the facial subspace regions and the color distribution features between the RGB image channels, and combining the image neighborhood feature relationship and the color distribution features between RGB, to perform effective feature extraction and classification, which can improve the accuracy of tampering detection of the target video;

[0115] (4) The number of parameters in this system is only a few tens of thousands, which is a fraction or even a few tens of times smaller than the number of parameters in many Deepfake models with comparable identification accuracy. It is a lightweight video tampering detection system.

[0116] Fourth embodiment

[0117] The present invention further provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following is achieved:

[0118] Such as the Deepfake video tampering detection method mentioned above.

[0119] The processor and the memory may be connected via a bus or other means.

[0120] The memory is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device.

[0121] In some embodiments, the memory may include a memory remotely located relative to the processor, and the remote memory may be connected to the processor via a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0122] The non-transitory software program and instructions required to implement the Deepfake video tampering detection method of the above embodiment are stored in the memory, and when executed by the processor, the Deepfake video tampering detection method in the above embodiment is executed.

[0123] Fifth embodiment

[0124] The present application also provides a computer-readable storage medium storing computer-executable instructions for executing:

[0125] Such as the Deepfake video tampering detection method mentioned above.

[0126] The computer-readable storage medium stores computer-executable instructions, which are executed by a processor or a controller, for example, by a processor in the above electronic device embodiment, so that the processor can execute:

[0127] Such as the Deepfake video tampering detection method in the above embodiment.

[0128] Those skilled in the art will appreciate that all or some of the steps and systems in the methods disclosed above may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0129] Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is well known to those of ordinary skill in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing data, such as computer-readable instructions, data structures, program modules, or other data.

[0130] Computer storage media includes, but is not limited to RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired data and which can be accessed by a computer.

[0131] Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any data delivery media.

[0132] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0133] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A Deepfake video tampering detection method, characterized in that: The steps include: Filtering a frame image sequence from a target video, wherein each frame image in the frame image sequence contains a human face region; Dividing the facial region of a human face in each frame image into a plurality of sub-frame images, each of the sub-frame images representing one of the subspace regions divided from the facial region of the human face; Performing feature extraction on all the sub-frame images through a preset cascade network; wherein, performing feature extraction on the sub-frame images through the cascade network comprises: Convolution is performed on the subframe image, pixel neighborhood correlation feature extraction is performed on the result after convolution, the feature extraction result is pooled, and the pooling result is characterized to obtain first feature map components of high frequency, medium frequency and low frequency; convolution is performed on the first feature map component of low frequency, pixel neighborhood correlation feature extraction is performed on the result after convolution, the feature extraction result is pooled, and the pooling result is characterized to obtain second feature map components of high frequency, medium frequency and low frequency; and so on, until convolution is performed on the N-1th feature map component of low frequency, pixel neighborhood correlation feature extraction is performed on the result after convolution, the feature extraction result is pooled, and the pooling result is characterized to obtain Nth feature map component of high frequency, medium frequency and low frequency; wherein N refers to the maximum number of layers of the cascade network; Generate a feature vector according to the first feature map component to the Nth feature map component of all intermediate frequencies; the generating a feature vector according to the first feature map component to the Nth feature map component of all intermediate frequencies comprises the steps of: Perform principal component analysis on the first to Nth feature map components of the intermediate frequency by PCA and retain the set energy value to generate a feature vector; Video tampering detection is performed according to the feature vector.

2. The Deepfake video tampering detection method according to claim 1, characterized in that: The step of selecting a frame image sequence from a target video comprises the following steps: Selecting all frame images containing human face areas from the target video in sequence; Calculating the pixel matrix distance between two adjacent frame images, when the pixel matrix distance between two adjacent frame images is greater than a threshold, retaining the two adjacent frame images; when the pixel matrix distance between two adjacent frame images is less than or equal to the threshold, deleting the frame image with lower definition among the two adjacent frame images; The retained frame images are combined into a frame image sequence.

3. The Deepfake video tampering detection method according to claim 2, characterized in that: The calculating of the pixel matrix distance between two adjacent frame images comprises: The color frame image is converted into a grayscale frame image, and the pixel matrix distance between two adjacent grayscale frame images is calculated.

4. The Deepfake video tampering detection method according to claim 1, characterized in that: Before dividing the facial area of ​​each frame image into a plurality of sub-frame images, the method further includes the following steps: When the size of the face area of ​​the frame image is smaller than a preset value, the frame image smaller than the preset value is super-resolutioned through the RCAN network.

5. The Deepfake video tampering detection method according to claim 1, characterized in that: After generating the feature vector, the method further includes: The feature vector is assigned a classification label through the XGBoost classifier.

6. The Deepfake video tampering detection method according to any one of claims 1 to 5, characterized in that: The cascade network consists of 4 layers of sub-networks, the convolution kernel size of the first two layers of sub-networks is 2*2, and the step size is 1; the convolution kernel size of the last two layers of sub-networks is 3*3, and the step size is 1.

7. A Deepfake video tampering detection system, characterized in that: include: A frame image screening module, used for screening out a frame image sequence from a target video, wherein each frame image in the frame image sequence contains a human face area; A subspace division module, used for dividing the face area of ​​each frame image into a plurality of subframe images, each of which represents one of the subspace areas divided from the face area; A cascade network module is used to extract features from all the sub-frame images, wherein extracting features from the sub-frame images through the cascade network includes: Convolution is performed on the subframe image, pixel neighborhood correlation feature extraction is performed on the result after convolution, the feature extraction result is pooled, and the pooling result is characterized to obtain first feature map components of high frequency, medium frequency and low frequency; convolution is performed on the first feature map component of low frequency, pixel neighborhood correlation feature extraction is performed on the result after convolution, the feature extraction result is pooled, and the pooling result is characterized to obtain second feature map components of high frequency, medium frequency and low frequency; and so on, until convolution is performed on the N-1th feature map component of low frequency, pixel neighborhood correlation feature extraction is performed on the result after convolution, the feature extraction result is pooled, and the pooling result is characterized to obtain Nth feature map component of high frequency, medium frequency and low frequency; wherein N refers to the maximum number of layers of the cascade network; A feature extraction module is used to generate a feature vector according to the first feature map component to the Nth feature map component of all intermediate frequencies; the feature vector is generated according to the first feature map component to the Nth feature map component of all intermediate frequencies, comprising the steps of: Perform principal component analysis on the first to Nth feature map components of the intermediate frequency by PCA and retain the set energy value to generate a feature vector; A decision module is used to perform video tampering detection according to the feature vector.

8. A Deepfake video tampering detection device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the Deepfake video tampering detection method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the Deepfake video tampering detection method as described in any one of claims 1 to 6.

Images