Deep fake face video detection method based on high-frequency features and visual Transform

Abstract

The invention discloses a deep fake face video detection method based on high-frequency features and a visual Transform, and the method comprises the steps: carrying out the preprocessing of face video data, and carrying out the high-frequency feature extraction of the obtained preprocessed image data; dividing the high-frequency features into feature blocks with fixed sizes, linearly flattening each feature block to obtain a feature block sequence, adding category token information to each feature block sequence and superposing relative position coding information, and then inputting the feature block sequence added with the category token information and the relative position coding information into a Transformer encoder for feature re-learning; and then analyzing and classifying the re-learning features to obtain a result whether the face in the face video data is a fake face. According to the method, the high-frequency features beneficial to counterfeit identification in the face video are extracted, and the visual Transform module is adopted to carry out feature re-learning on the high-frequency features, so that redundant feature extraction of the face video can be reduced, the utilization degree of cross-pixel relation information can be improved, and the detection performance of the deep counterfeit video can be optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of deep fake face video detection, and in particular to a deep fake face video detection method based on high-frequency features and a visual Transformer. Background Art

[0002] Deepfake refers to fake content generated using deep learning technology, mainly through complex neural network architectures such as generative adversarial networks (GANs). Deepfake face video detection is a technology that identifies videos that have been tampered with by artificial intelligence technology to make faces appear unreal.

[0003] Deep fake face video detection can be applied to the following areas:

[0004] 1. Media and News Industry

[0005] Used to verify the authenticity of the content.

[0006] In news dissemination, video is an important carrier of information. Deep fake face video detection can help media organizations ensure the authenticity of news materials. Through detection technology, news organizations can identify some fake videos, avoid spreading false information, and maintain the credibility of the news industry.

[0007] For copyright protection.

[0008] The works of media content creators may be infringed by deep fake technology. For example, faces in some film and television clips may be replaced to create spoof videos or unauthorized derivative works. Detection technology can help copyright owners discover infringing deep fake videos and safeguard their legitimate rights and interests.

[0009] 2. Judicial field

[0010] For review of evidence.

[0011] In the judicial process, the authenticity of video evidence is of vital importance. Deep fake face videos may be used to fake crime scenes or witness testimony. Through detection technology, judicial personnel can determine whether video evidence has been tampered with to ensure the fairness of judicial trials. For example, in cases involving identity recognition, such as fraud and theft, detecting whether the faces in the video are real can provide a reliable basis for the investigation and trial of the case.

[0012] Used to combat cyber crime.

[0013] Deep fake technology is often used by cyber criminals to conduct fraudulent activities, such as creating fake bank customer service videos to trick users into transferring money. Detection technology can help law enforcement agencies identify these fraudulent videos, thereby effectively combating cybercrime and protecting citizens' property safety.

[0014] 3. Social Networking Platforms

[0015] For information regulation.

[0016] There is a lot of user-generated content on social networks, which may include deep fake videos. Platforms can use detection technology to review videos uploaded by users to prevent the spread of false information and create a healthy online social environment. For example, in the dissemination of videos on some hot topics, detection technology can avoid social panic or cyber violence caused by false videos.

[0017] For user protection.

[0018] Ordinary users may be threatened by deep fake videos, such as their personal images being maliciously forged for bad purposes. Social platforms can protect users' privacy and reputation through detection technology, and promptly detect and handle deep fake videos involving users.

[0019] Currently, in deep fake face video detection, there are problems such as insufficient extraction and utilization of frequency domain features of fake video faces, especially high-frequency features, as well as insufficient extraction and utilization of cross-pixel relationship information. Summary of the invention

[0020] In view of the shortcomings of the prior art, the present invention aims to provide a deep fake face video detection method based on high-frequency features and visual Transformer.

[0021] In order to achieve the above object, the present invention adopts the following technical solution:

[0022] A deep fake face video detection method based on high-frequency features and visual Transformer. The specific process is as follows:

[0023] S1, preprocessing: extract frames, locate faces and enhance images of face video data in sequence;

[0024] S2, extracting high-frequency features from the preprocessed image data obtained in step S1;

[0025] S3, dividing the high-frequency features obtained in step S2 into feature blocks of fixed size, linearly flattening each feature block to obtain a feature block sequence, adding category token information to each feature block sequence and superimposing relative position encoding information, and then inputting the feature block sequence after adding category token information and relative position encoding information into the Transformer encoder for feature re-learning;

[0026] S4. Analyze and classify the relearned features obtained in step S3 to determine whether the face in the face video data is a forged face.

[0027] Furthermore, the specific process of step S2 is: firstly, Fourier transform is used to obtain frequency domain information of the image data, wherein the frequency domain information includes high frequency information and low frequency information; and then a high pass filter is used to remove the low frequency information in the frequency domain information and retain the high frequency features.

[0028] Furthermore, the specific process of step S3 is as follows: relative position encoding uses the relative distance and pair relationship between different feature blocks to supplement the position information of the feature blocks in the original image data, and is calculated through the query representation that is the same as the query and key in the self-attention mechanism, as shown in formula (1):

[0029]

[0030] In formula (1), d z represents the dimension of the sequence z, and represents the learnable vector, x i and x j represents the i-th token and the j-th token, W Q , W K are the parameter matrices of the query and key subspaces in the self-attention mechanism, respectively; Indicates the encoding of the pixel position in the x direction, Indicates the encoding of the pixel position in the y direction, and represents the components of the i-th vector in the x and y directions, and represents the components of the jth vector in the x and y directions, is the two-dimensional coordinate of the relative position, g(x) is used to index the relative distance of the corresponding code x; [·] indicates rounding operation; sign(·) determines the sign of the number, positive input returns 1, negative input returns -1, and vice versa returns 0; α determines the segmentation point, β controls the output range to [-β, β], and γ adjusts the curvature of the logarithmic part;

[0031] In the self-attention mechanism, query, key and value are all vectors, representing query, key and value respectively, which are linear transformation representations of feature block sequences of high-frequency features in three spatial dimensions; given a query vector, one or more related value vectors are obtained by matching with the key vector; query is the vector to be searched input in the attention mechanism, which is the information to be queried; when using the self-attention mechanism for calculation, the similarity between query and key is calculated to determine the weight of value in weighted averaging; key is a set of vectors used for search; key vectors are vectors associated with query vectors, which are used to describe the information in the input sequence; when calculating the similarity between query and key, all vectors in the key set are used as the search space and the vector most similar to the query vector is found; value is the vector corresponding to the key vector; value vector is the vector associated with each key, representing the information at the corresponding position in the input sequence; when weighted averaging, the similarity between query and key vectors is used as the weight, and the corresponding value vectors are weighted and summed, as shown in the following formula:

[0032] h i =Attention(W i q q,W i k k,W i v v) (2)

[0033] Among them, q, k, and v represent query, key, and value vectors. are the linear transformation weights of query, key and value vectors respectively, d model Represents the dimension of the feature block sequence, d q d k d v Represent the dimensions of query, key and value vectors respectively;

[0034] In the Transformer encoder, q, k and v are mapped to multiple representation subspaces through linear transformation; then a multi-head attention mechanism is used in each representation subspace to obtain the corresponding results, and finally all the outputs are spliced ​​together, and then a linear transformation is performed to obtain the final result, which is then input into a multi-layer perceptron for nonlinear transformation and feature extraction integration to obtain the re-learning features.

[0035] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0036] The present invention also provides a computer device, comprising a processor and a memory, wherein the memory is used to store a computer program; and when the processor is used to execute the computer program, the above method is implemented.

[0037] The beneficial effects of the present invention are as follows: the present invention can reduce redundant feature extraction of face videos, improve the utilization of cross-pixel relationship information, and optimize deep fake video detection performance by extracting high-frequency features in face videos that are helpful for forgery identification, and using a visual Transformer module to perform feature relearning on the high-frequency features. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 The overall architecture diagram of the method for implementing the embodiment of the present invention;

[0039] Figure 2 Schematic diagram of the architecture of the Transformer encoder in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described below in conjunction with the accompanying drawings. It should be noted that this embodiment is based on the technical solution and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to this embodiment.

[0041] This embodiment provides a deep fake face video detection method based on high-frequency features and visual Transformer. Figure 1 As shown, the specific process is:

[0042] S1. Preprocessing: extract frames, locate faces and enhance images of face video data in sequence.

[0043] S2. Extract high-frequency features from the preprocessed image data obtained in step S1: first use Fourier transform to obtain frequency domain information of the image data, where the frequency domain information includes high-frequency information and low-frequency information; then use a high-pass filter to remove the low-frequency information in the frequency domain information and retain the high-frequency features.

[0044] S3. Divide the high-frequency features obtained in step S2 into feature blocks of fixed size, linearly flatten each feature block to obtain a feature block sequence, add category token information to each feature block sequence and superimpose relative position encoding information, and then input the feature block sequence after adding category token information and relative position encoding information into the Transformer encoder for feature re-learning.

[0045] Relative Position Encoding (RPE) uses the relative distance and pairwise relationship between different feature blocks to supplement the position information of feature blocks in the original image data, which can be calculated by the same query representation as the query and key in the self-attention mechanism, as shown in formula (1):

[0046]

[0047] In formula (1), d z represents the dimension of the sequence z, and represents the learnable vector, x i and x j represents the i-th token and the j-th token, W Q , W K are the parameter matrices of the query and key subspaces in the self-attention mechanism, respectively. Indicates the encoding of the pixel position in the x direction, Indicates the encoding of the pixel position in the y direction, and represents the components of the i-th vector in the x and y directions, and represents the components of the jth vector in the x and y directions, is the two-dimensional coordinate of the relative position, g(x) is used to index the relative distance of the corresponding code x. [·] indicates rounding operation, sign(·) determines the sign of the number (positive input returns 1, negative input returns -1, and vice versa returns 0), α determines the segmentation point, β controls the output range to [-β, β], and γ adjusts the curvature of the logarithmic part.

[0048] In the self-attention mechanism, query, key and value are all vectors, representing query, key and value respectively. In this embodiment, query, key and value vectors are linear transformation representations of feature block sequences of high-frequency features in three spatial dimensions. They can be regarded as a set of mapping relationships: given a query vector, one or more related value vectors are obtained by matching with the key vector. Specifically, query is the vector to be searched input in the attention mechanism, which is the information to be queried. When using the self-attention mechanism for calculation, the similarity between query and key is calculated to determine the weight of value in weighted averaging. Key is a set of vectors used for search. Key vectors are some vectors associated with query vectors, which are used to describe the information in the input sequence. When calculating the similarity between query and key, all vectors in the key set can be used as the search space and the vector most similar to the query vector can be found. Value is the vector corresponding to the key vector. The value vector is a vector associated with each key, representing the information at the corresponding position in the input sequence. When weighted averaging, the similarity between query and key vectors is used as the weight, and the corresponding value vectors are weighted and summed, as shown in the following formula:

[0049] h i =Attention(W i q q,W i k k,W i v v) (2)

[0050] Among them, q, k, and v represent query, key, and value vectors. are the linear transformation weights of query, key and value vectors respectively, d model Represents the dimension of the feature block sequence, d q d k d v Represents the dimensions of query, key and value vectors respectively.

[0051] The Multi-Head Attention mechanism is based on the Scaled Dot-Product Attention (SA) mechanism. The core idea of ​​SA is to iteratively update the sequence representation based on the correlation between sequence elements. Figure 2As shown in the figure, in the Transformer encoder, q, k and v are mapped to multiple representation subspaces through linear transformation; then a multi-head attention mechanism is used in each representation subspace to obtain the corresponding results, and finally all the outputs are spliced ​​together, and then a linear transformation is performed to obtain the final result, which is then input into a multilayer perceptron (MLP) for nonlinear transformation and feature extraction integration to obtain the re-learning features.

[0052] S4. Input the relearned features obtained in step S3 into a multi-layer perceptron head (MLP Head) for analysis and classification to obtain a result of whether the face in the face video data is a forged face.

[0053] Because the shape of the Transformer encoder output and input is consistent, in practical applications, MLPHead generally directly uses a softmax function for classification.

[0054] For those skilled in the art, various corresponding changes and modifications can be made according to the above technical solutions and concepts, and all of these changes and modifications should be included in the protection scope of the claims of the present invention.

Claims

1. A deep fake face video detection method based on high-frequency features and visual Transformer, characterized in that: The specific process is: S1, preprocessing: extract frames, locate faces and enhance images of face video data in sequence; S2, extracting high-frequency features from the preprocessed image data obtained in step S1; S3, dividing the high-frequency features obtained in step S2 into feature blocks of fixed size, linearly flattening each feature block to obtain a feature block sequence, adding category token information to each feature block sequence and superimposing relative position encoding information, and then inputting the feature block sequence after adding category token information and relative position encoding information into the Transformer encoder for feature re-learning; S4. Analyze and classify the relearned features obtained in step S3 to determine whether the face in the face video data is a forged face.

2. The method according to claim 1, characterized in that The specific process of step S2 is: firstly, Fourier transform is used to obtain the frequency domain information of the image data, wherein the frequency domain information includes high frequency information and low frequency information; then a high pass filter is used to remove the low frequency information in the frequency domain information and retain the high frequency features.

3. The method according to claim 1, characterized in that The specific process of step S3 is as follows: relative position encoding uses the relative distance and pairwise relationship between different feature blocks to supplement the position information of the feature blocks in the original image data, and is calculated through the query representation that is the same as the query and key in the self-attention mechanism, as shown in formula (1): In formula (1), d z represents the dimension of the sequence z, and represents the learnable vector, x i and x j represents the i-th token and the j-th token, W Q , W K are the parameter matrices of the query and key subspaces in the self-attention mechanism, respectively; Indicates the encoding of the pixel position in the x direction, Indicates the encoding of the pixel position in the y direction, and represents the components of the i-th vector in the x and y directions, and represents the components of the jth vector in the x and y directions, is the two-dimensional coordinate of the relative position, g(x) is used to index the relative distance of the corresponding code x; [·] indicates rounding operation; sign(·) determines the sign of the number, positive input returns 1, negative input returns -1, and vice versa returns 0; α determines the segmentation point, β controls the output range to [-β, β], and γ adjusts the curvature of the logarithmic part; In the self-attention mechanism, query, key and value are all vectors, representing query, key and value respectively, which are linear transformation representations of feature block sequences of high-frequency features in three spatial dimensions; given a query vector, one or more related value vectors are obtained by matching with the key vector; query is the vector to be searched input in the attention mechanism, which is the information to be queried; when using the self-attention mechanism for calculation, the similarity between query and key is calculated to determine the weight of value in weighted averaging; key is a set of vectors used for search; key vectors are vectors associated with query vectors, which are used to describe the information in the input sequence; when calculating the similarity between query and key, all vectors in the key set are used as the search space and the vector most similar to the query vector is found; value is the vector corresponding to the key vector; value vector is the vector associated with each key, representing the information at the corresponding position in the input sequence; when weighted averaging, the similarity between query and key vectors is used as the weight, and the corresponding value vectors are weighted and summed, as shown in the following formula: h i =Attention(W i q q,W i k k,W i v v) (2) Among them, q, k, and v represent query, key, and value vectors. are the linear transformation weights of query, key and value vectors respectively, d model Represents the dimension of the feature block sequence, d q ,d k ,d v Represent the dimensions of query, key and value vectors respectively; In the Transformer encoder, q, k and v are mapped to multiple representation subspaces through linear transformation; then a multi-head attention mechanism is used in each representation subspace to obtain the corresponding results, and finally all the outputs are spliced ​​together, and then a linear transformation is performed to obtain the final result, which is then input into a multi-layer perceptron for nonlinear transformation and feature extraction integration to obtain the re-learning features.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

5. A computer device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store a computer program; when the processor is used to execute the computer program, the method described in any one of claims 1 to 3 is implemented.

Images