Method, device, processor and computer storage medium for unsupervised deep fake video detection based on error reconstruction

Abstract

The present invention relates to a method for unsupervised deep fake video detection based on error reconstruction, wherein the method comprises: (1) inputting a video to be detected for data preprocessing, and extracting a facial feature point sequence in the video to be detected; (2) reconstructing the facial feature point sequence to generate a facial feature point reconstruction sequence; (3) calculating the fake score of the current video based on the facial feature point sequence and the facial feature point reconstruction sequence; (4) judging the authenticity of the current video based on the comparison relationship between the fake score and a preset threshold. The present invention also relates to a corresponding device, a processor and a computer-readable storage medium thereof. The method, device, processor and computer-readable storage medium thereof for unsupervised deep fake video detection based on error reconstruction of the present invention do not need to use fake videos for training, save the cost of generating fake videos, and effectively balance performance and usage cost.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to the field of deep fake video detection technology, and specifically refers to a method, device, processor and computer-readable storage medium thereof for implementing unsupervised deep fake video detection processing based on error reconstruction. Background Art

[0002] With the rapid development of deep generation technology, the level of automatic content generation by machines has been continuously improved. Deepfake is a popular technology in content generation and has been widely used in short videos, game live broadcasts, video conferencing and other fields. However, deepfakes with extremely high deceptiveness have caused many controversies and brought potential social trust and social security issues.

[0003] Currently, forged video detection is mainly modeled as a supervised classification task, including single-frame-based classification detection and multi-frame-based classification detection. The former mainly uses features such as abnormal color, eye and tooth details to detect forged videos in a single frame; the latter models the timing information such as blinking frequency and expression changes in the video. Compared with single-frame methods, the performance of multi-frame-based methods has been improved to varying degrees, proving the importance of introducing timing information to improve detection accuracy.

[0004] However, there are still two major problems in the current work: First, due to the continuous iteration and update of deep fake methods, there are many types, and the existing supervised detection algorithm training relies on the labeled data, which requires the generation of corresponding fake models. Most of these fake models are not open source, difficult to reproduce, and have high training costs, which greatly increases the training and update costs of the detection model; second, the method based on deep convolutional network feature extraction has good performance but high training costs, which further increases the application threshold of the fake detection model. Therefore, a deep fake detection method that balances performance and cost is needed. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a method, device, processor and computer-readable storage medium for unsupervised deep fake video detection processing based on error reconstruction, which can effectively achieve a good balance between performance and cost.

[0006] In order to achieve the above objectives, the method, device, processor and computer-readable storage medium thereof for implementing unsupervised deep fake video detection processing based on error reconstruction of the present invention are as follows:

[0007] The method for realizing unsupervised deep fake video detection and processing based on error reconstruction is mainly characterized in that the method comprises the following steps:

[0008] (1) inputting a video to be detected for data preprocessing, and extracting a sequence of facial feature points in the video to be detected;

[0009] (2) reconstructing the facial feature point sequence to generate a facial feature point reconstructed sequence;

[0010] (3) calculating the forgery score of the current video according to the facial feature point sequence and the facial feature point reconstruction sequence;

[0011] (4) The authenticity of the current video is determined based on a comparison between the forgery score and a preset threshold.

[0012] Preferably, the step (1) comprises the following steps:

[0013] (1.1) Using the facial feature point extraction tool Dlib, 68 facial feature points of each frame image in the video to be detected are extracted as the original features of the frame;

[0014] (1.2) The original facial feature point sequence is obtained by combining the original video frames in order, denoted as f seq =[f1, f2, f3, ..., f N ] Where N is the number of video frames, f is the facial feature points of the corresponding frame; the 68 facial feature points of the i-th frame are expressed as in They are respectively represented as the horizontal and vertical coordinates of the jth feature point;

[0015] (1.2) The original facial feature point sequence of each frame is processed by Lucas-Kanade optical flow algorithm to predict the next frame to obtain a predicted frame;

[0016] (1.3) Each frame of the original facial feature point sequence and the corresponding predicted frame are merged through a Kalman filter, and the noise in the original facial feature point sequence is removed to extract the denoised facial feature point sequence.

[0017] Preferably, the step (2) comprises the following steps:

[0018] (2.1) extracting local features of the facial feature point sequence using a convolutional neural network;

[0019] (2.2) Then use the bidirectional gated recurrent network to model the features of each frame sequence of the facial feature point sequence to obtain a deep coding representation sequence Z seq =[z1,z2,z3,...,z N ], N is the number of sequence codes;

[0020] (2.3) The deep coding sequence Z seq =[z1,z2,z3,...,z N ] through two independent fully connected layers to perform parameter processing and obtain the desired μ i and variance σ i , that is, μ i =fc μ (z i ), σ i =fc σ (z i ), where fc u and fc σ are independent fully connected layers, and the encoding representation sequence after re-parameterization is composed of the expected μ i , variance σ i and sampling from a normal distribution ξ i Calculated, specifically:

[0021] (2.4) Using bidirectional gated recurrent neural network and fully connected network to represent the encoded sequence after re-parameterization After decoding, the reconstructed feature sequence is obtained as f′ seq =[f′1, f′2, f′3,..., f′ N ].

[0022] Preferably, the step (3) comprises the following steps:

[0023] (3.1) Calculate the reconstruction error of the facial feature point sequence and the reconstructed feature sequence for each frame, that is, the reconstruction error of the i-th frame is in They are respectively represented as the horizontal and vertical coordinates of the jth original feature point, They are respectively represented as the horizontal and vertical coordinates of the j-th reconstructed feature point;

[0024] (3.2) The reconstruction errors of each frame are accumulated, and the sum of the reconstruction errors is used as the forgery score of the current video, that is,

[0025] Preferably, the step (4) comprises the following steps:

[0026] (4.1) Setting a preset threshold for judging the authenticity of a video;

[0027] (4.2) Compare the calculated forgery score with the preset threshold. If the forgery score is greater than the preset threshold, the current video is determined to be a forged video; if the forgery score is less than the preset threshold, the current video is determined to be a normal video.

[0028] The main feature of the device for realizing unsupervised deep fake video detection processing based on error reconstruction is that the device comprises:

[0029] a processor configured to execute computer executable instructions;

[0030] A memory stores one or more computer executable instructions. When the computer executable instructions are executed by the processor, the various steps of the method for unsupervised deep fake video detection processing based on error reconstruction are implemented.

[0031] The processor for implementing unsupervised deep fake video detection and processing based on error reconstruction has the main feature that the processor is configured to execute computer-executable instructions. When the computer-executable instructions are executed by the processor, the various steps of the method for implementing unsupervised deep fake video detection and processing based on error reconstruction are implemented.

[0032] The main feature of this computer-readable storage medium is that a computer program is stored thereon, and the computer program can be executed by a processor to implement the various steps of the above-mentioned method for unsupervised deep fake video detection processing based on error reconstruction.

[0033] The method, device, processor and computer storage medium thereof for implementing unsupervised deep fake video detection and processing based on error reconstruction of the present invention are suitable for deep fake video detection in low-resource scenarios. Its working characteristics differ from those of existing deep fake video detection methods in that it uses facial feature points as basic features, reconstructs the sequence using an unsupervised method, and performs authenticity detection based on the reconstruction error. At the same time, the model constructed by the present technical solution does not need to use fake videos for training, which saves the cost of generating fake videos and the update cost required for the detection model when new fake methods appear; using facial feature points as basic features can greatly reduce the feature space while retaining key fake features, thereby effectively balancing performance and cost and reducing the cost of model application. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a schematic diagram of the overall process of the method for unsupervised deep fake video detection and processing based on error reconstruction of the present invention.

[0035] Figure 2A schematic diagram of the network structure of an improved variational autoencoder according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to more clearly describe the technical content of the present invention, further description is given below in conjunction with specific embodiments.

[0037] Before describing in detail embodiments according to the present invention, it should be noted that, hereinafter, the terms "comprises", "includes" or any other variations are intended to cover non-exclusive inclusion, whereby a process, method, article or apparatus comprising a series of elements includes not only these elements, but also other elements not explicitly listed or inherent to such process, method, article or apparatus.

[0038] See also Figure 1 As shown, the method for unsupervised deep fake video detection and processing based on error reconstruction, wherein the method comprises the following steps:

[0039] (1) inputting a video to be detected for data preprocessing, and extracting a sequence of facial feature points in the video to be detected;

[0040] (2) reconstructing the facial feature point sequence to generate a facial feature point reconstructed sequence;

[0041] (3) calculating the forgery score of the current video according to the facial feature point sequence and the facial feature point reconstruction sequence;

[0042] (4) The authenticity of the current video is determined based on a comparison between the forgery score and a preset threshold.

[0043] As a preferred embodiment of the present invention, the step (1) comprises the following steps:

[0044] (1.1) Using the facial feature point extraction tool Dlib, 68 facial feature points of each frame image in the video to be detected are extracted as the original features of the frame;

[0045] (1.2) The original facial feature point sequence is obtained by combining the original video frames in order, denoted as f seq =[f1,f2,f3,...,f N ] Where N is the number of video frames, f is the facial feature points of the corresponding frame; the 68 facial feature points of the i-th frame are expressed as in They are respectively represented as the horizontal and vertical coordinates of the jth feature point;

[0046] (1.2) The original facial feature point sequence of each frame is processed by Lucas-Kanade optical flow algorithm to predict the next frame to obtain a predicted frame;

[0047] (1.3) Each frame of the original facial feature point sequence and the corresponding predicted frame are merged through a Kalman filter, and the noise in the original facial feature point sequence is removed to extract the denoised facial feature point sequence.

[0048] As a preferred embodiment of the present invention, the step (2) comprises the following steps:

[0049] (2.1) extracting local features of the facial feature point sequence using a convolutional neural network;

[0050] (2.2) Then use the bidirectional gated recurrent network to model the features of each frame sequence of the facial feature point sequence to obtain a deep coding representation sequence Z seq =[z1,z2,z3,...,z N ], N is the number of sequence codes;

[0051] (2.3) The deep coding sequence Z seq =[z1,z2,z3,...,z N ] through two independent fully connected layers to perform parameter processing and obtain the desired μ i and variance σ i , that is, μ i =fc μ (z i ), σ i =fc σ (z i ), where fc u and fc σ are independent fully connected layers, and the encoding representation sequence after re-parameterization is composed of the expected μ i , variance σ i and sampling from a normal distribution ξ i Calculated, specifically:

[0052] (2.4) Using bidirectional gated recurrent neural network and fully connected network to represent the encoded sequence after re-parameterization After decoding, the reconstructed feature sequence is obtained as f′ seq =[f′1, f′2, f′3,..., f′ N ].

[0053] As a preferred embodiment of the present invention, the step (3) comprises the following steps:

[0054] (3.1) Calculate the reconstruction error of the facial feature point sequence and the reconstructed feature sequence for each frame, that is, the reconstruction error of the i-th frame is in They are respectively represented as the horizontal and vertical coordinates of the jth original feature point, They are respectively represented as the horizontal and vertical coordinates of the j-th reconstructed feature point;

[0055] (3.2) The reconstruction errors of each frame are accumulated, and the sum of the reconstruction errors is used as the forgery score of the current video, that is,

[0056] As a preferred embodiment of the present invention, the step (4) comprises the following steps:

[0057] (4.1) Setting a preset threshold for judging the authenticity of a video;

[0058] (4.2) Compare the calculated forgery score with the preset threshold. If the forgery score is greater than the preset threshold, the current video is determined to be a forged video; if the forgery score is less than the preset threshold, the current video is determined to be a normal video.

[0059] In the practical application of the present invention, the technical solution includes three stages: data preprocessing, generation of facial feature point reconstruction sequence, and calculation of forgery score. When a user wants to detect the authenticity of a video, the system first preprocesses the video to obtain the facial feature point sequence corresponding to the video, then generates the facial feature point reconstruction sequence through the improved variational autoencoder, and then calculates the forgery score through the original facial feature point sequence and the reconstructed sequence, and finally determines whether the video is forged based on whether the forgery score exceeds the set threshold.

[0060] In one embodiment of the present invention, please refer to Figure 1 The overall processing flow of this technical solution is shown below:

[0061] 1) Input the video to be detected.

[0062] 2) Obtain the facial feature point sequence of the video to be detected through data preprocessing.

[0063] 3) Generate a facial feature point reconstruction sequence based on the facial feature point sequence through feature point sequence reconstruction.

[0064] 4) Calculate the forgery score based on the facial feature point sequence and the facial feature point reconstruction sequence.

[0065] 5) Determine whether the forgery score exceeds the set threshold. If it exceeds, it is forgery; if it does not exceed, it is normal.

[0066] In one embodiment of the present invention, please refer to Figure 2 As shown, it is the network structure of the variational autoencoder improved by the present technical solution.

[0067] For an input sequence of N frames of facial feature points, the jth facial feature point in the i-th frame is marked as i∈[0,N],j∈[1,68],each feature point The horizontal and vertical axes are expressed as The feature of the i-th frame can be expressed as Then, a convolutional neural network is used to extract local features between frames, and then a bidirectional gated recurrent neural network is used to model the sequence features of the facial feature point sequence. Specifically, the feature sequence is represented as f seq =[f1, f2, f3, ..., f N ], after passing through the convolutional neural network and the bidirectional gated recurrent neural network in turn, the deep encoding sequence is Z seq =[z1,z2,z3,...,z N ]Since the gradient cannot be transferred during the sampling training process, the re-parameter method is adopted. That is, for the deep encoding representation z i , i∈[0,N], the expectation and variance are obtained through two independent fully connected layers, and the encoding after re-parameterization is expressed as: During the decoding process, the temporal information of the previous and next codes also needs to be considered, and a bidirectional gated recurrent neural network and a fully connected network are used to decode in sequence.

[0068] Finally, the reconstructed feature sequence is f′ seq =[f′1, f′2, f′3,..., f′ N ], the loss function consists of two parts: the reconstruction error of each facial feature point and the KL divergence, namely:

[0069] loss=loss reco +w×KL(μ,σ)

[0070]

[0071] KL(μ,σ)=-0.5×(1+log σ 2 -μ 2 -exp(log 2 ))

[0072] where w is the coefficient of KL divergence. They are respectively represented as the horizontal and vertical coordinates of the jth original feature point, They are respectively represented as the horizontal and vertical coordinates of the jth reconstructed feature point. μ and σ correspond to the expected μ generated by resampling. i, variance σ i The mean of .

[0073] The device for realizing unsupervised deep fake video detection processing based on error reconstruction, wherein the device comprises:

[0074] a processor configured to execute computer executable instructions;

[0075] A memory stores one or more computer executable instructions. When the computer executable instructions are executed by the processor, the various steps of the method for unsupervised deep fake video detection processing based on error reconstruction are implemented.

[0076] The processor for implementing unsupervised deep fake video detection processing based on error reconstruction, wherein the processor is configured to execute computer-executable instructions, and when the computer-executable instructions are executed by the processor, the various steps of the method for implementing unsupervised deep fake video detection processing based on error reconstruction are implemented.

[0077] The computer-readable storage medium stores a computer program thereon, and the computer program can be executed by a processor to implement the various steps of the method for unsupervised deep fake video detection processing based on error reconstruction as described above.

[0078] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0079] It should be understood that each part of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device.

[0080] A person of ordinary skill in the art may understand that all or part of the steps of the method for implementing the above-mentioned embodiment may be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0081] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0082] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0083] In the description of this specification, the description with reference to the terms "an embodiment", "some embodiments", "example", "specific example", or "embodiment" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0084] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

[0085] The method, device, processor and computer storage medium thereof for implementing unsupervised deep fake video detection and processing based on error reconstruction of the present invention are suitable for deep fake video detection in low-resource scenarios. Its working characteristics differ from those of existing deep fake video detection methods in that it uses facial feature points as basic features, reconstructs the sequence using an unsupervised method, and performs authenticity detection based on the reconstruction error. At the same time, the model constructed by the present technical solution does not need to use fake videos for training, which saves the cost of generating fake videos and the update cost required for the detection model when new fake methods appear; using facial feature points as basic features can greatly reduce the feature space while retaining key fake features, thereby effectively balancing performance and cost and reducing the cost of model application.

[0086] In this specification, the present invention has been described with reference to specific embodiments thereof. However, it is apparent that various modifications and variations may be made without departing from the spirit and scope of the present invention. Therefore, the specification and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A method for unsupervised deep fake video detection based on error reconstruction, characterized in that: The method comprises the following steps: (1) inputting a video to be detected for data preprocessing, and extracting a sequence of facial feature points in the video to be detected; (2) reconstructing the facial feature point sequence to generate a facial feature point reconstructed sequence; (3) calculating the forgery score of the current video according to the facial feature point sequence and the facial feature point reconstruction sequence; (4) judging the authenticity of the current video based on a comparison between the forgery score and a preset threshold; The step (2) comprises the following steps: (2.1) extracting local features of the facial feature point sequence using a convolutional neural network; (2.2) Then use the bidirectional gated recurrent network to model the features of each frame sequence of the facial feature point sequence to obtain a deep coding representation sequence Z seq =[z1,z2,z3,…,z N ], N is the number of sequence codes; (2.3) The deep coding representation sequence Z seq =[z1,z2,z3,…,z N ] Through two independent fully connected layers, the parameters are reprocessed to obtain the desired μ i and variance σ i , that is, μ i =fc μ (z i ),σ i =fc σ (z i ), where fc u and fc σ are independent fully connected layers, and the encoding representation sequence after re-parameterization is composed of the expected μ i , variance σ i and sampling from a normal distribution ξ i Calculated, specifically: ξ i ~N(0,I); (2.4) Using bidirectional gated recurrent neural network and fully connected network to represent the encoded sequence after re-parameterization After decoding, the reconstructed feature sequence is obtained as f′ seq =[f′1,f′2,f′3,…,f′ N ]; The step (3) comprises the following steps: (3.1) Calculate the reconstruction error of the facial feature point sequence and the reconstructed feature sequence for each frame, that is, the reconstruction error of the i-th frame is in They are respectively represented as the horizontal and vertical coordinates of the jth original feature point, They are respectively represented as the horizontal and vertical coordinates of the j-th reconstructed feature point; (3.2) The reconstruction errors of each frame are accumulated, and the sum of the reconstruction errors is used as the forgery score of the current video, that is, 2. The method for unsupervised deep fake video detection based on error reconstruction according to claim 1 is characterized in that: The step (1) comprises the following steps: (1.1) Using the facial feature point extraction tool Dlib, 68 facial feature points of each frame image in the video to be detected are extracted as the original features of the frame; (1.2) The original facial feature point sequence is obtained by combining the original video frames in order, denoted as f seq =[f1,f2,f3,…,f N ] Where N is the number of video frames, f is the facial feature point of the corresponding frame; the 68th facial feature point of the i-th frame is expressed as in They are respectively represented as the horizontal and vertical coordinates of the jth feature point; (1.3) The original facial feature point sequence of each frame is processed by Lucas-Kanade optical flow algorithm to predict the next frame to obtain a predicted frame; (1.4) Each frame of the original facial feature point sequence and the corresponding predicted frame are merged through a Kalman filter, and the noise in the original facial feature point sequence is removed to extract the denoised facial feature point sequence.

3. The method for unsupervised deep fake video detection based on error reconstruction according to claim 1 is characterized in that: The step (4) comprises the following steps: (4.1) Setting a preset threshold for judging the authenticity of a video; (4.2) Compare the calculated forgery score with the preset threshold. If the forgery score is greater than the preset threshold, the current video is determined to be a forged video; if the forgery score is less than the preset threshold, the current video is determined to be a normal video.

4. A device for realizing unsupervised deep fake video detection processing based on error reconstruction, characterized in that: The device comprises: a processor configured to execute computer-executable instructions; A memory storing one or more computer executable instructions, wherein when the computer executable instructions are executed by the processor, the steps of the method for unsupervised deep fake video detection based on error reconstruction described in any one of claims 1 to 3 are implemented.

5. A processor for unsupervised deep fake video detection based on error reconstruction, characterized in that: The processor is configured to execute computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method for unsupervised deep fake video detection based on error reconstruction described in any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the various steps of the method for unsupervised deep fake video detection processing based on error reconstruction as described in any one of claims 1 to 3.

Images