Deep fake video detection method and device based on new neural network architecture

Abstract

The present application relates to a method and device for detecting deep fake videos based on a new neural network architecture. The method comprises: constructing a new neural network structure; the new neural network architecture comprises a data enhancement module, an MTCNN algorithm module, a feature extractor, a linear layer, a fusion calculation layer, and a linear classifier; training the new neural network structure according to a pre-acquired training set, inputting the disguised video data to be detected into the trained new neural network, performing data enhancement on the disguised video data using the data enhancement module, identifying the enhanced disguised video data according to the MTCNN algorithm module, extracting and mapping features from the image data set using the feature extractor and the linear layer, performing feature fusion on the obtained feature vector set using the fusion calculation layer, classifying and detecting the fused features according to the linear classifier, and obtaining the detection results. The present method can improve the accuracy of deep fake video detection.

Description

Technical Field

[0001] The present application relates to the field of video detection technology, and in particular to a method and device for detecting deep fake videos based on a novel neural network architecture. Background Art

[0002] With innovative breakthroughs in deep learning algorithms, computer vision and imaging technologies are experiencing a new wave of development, making it increasingly easy to generate highly realistic facially altered images or videos. A new generation of generative deep neural networks can automatically generate highly realistic images or videos from large amounts of training data. The emergence of deepfakes, in particular, has significantly lowered the barrier to entry for facial alteration technology. Deepfakes use generative adversarial networks to nearly perfectly replace the face in the original video with that of another person. While deepfake technology is a double-edged sword, providing amazing entertainment, its misuse has also led to an increasing number of fake videos and news. The rapid rise in popularity of facial alteration applications has sparked concerns about the spread of fake images and videos. A growing number of researchers are devoting themselves to the field of facial alteration and forgery detection.

[0003] However, current methods of detecting fake videos by using model integration and knowledge distillation improve detection accuracy, but on the one hand, they make the model bloated and difficult to train. On the other hand, the model's generalization ability is very limited and it focuses on a low semantic level. Therefore, it is often ineffective in identifying deep fake videos with unknown methods and sources. Summary of the Invention

[0004] Based on this, it is necessary to provide a deep fake video detection method, device, computer equipment and storage medium based on a new neural network architecture that can improve the accuracy of deep fake video detection in response to the above technical problems.

[0005] A deep fake video detection method based on a novel neural network architecture, the method comprising:

[0006] Obtaining disguised video data to be detected;

[0007] Construct a new neural network structure; the new neural network architecture includes a data enhancement module, an MTCNN algorithm module, a feature extractor, a linear layer, a fusion calculation layer, and a linear classifier;

[0008] The new neural network structure is trained according to the pre-acquired training set to obtain a trained new neural network structure;

[0009] The disguised video data to be detected is input into the trained new neural network, and the disguised video data is enhanced using the data enhancement module to obtain enhanced disguised video data;

[0010] The enhanced disguised video data is identified according to the MTCNN algorithm module to obtain an image dataset;

[0011] Use the feature extractor to extract features from the image dataset to obtain a feature atlas;

[0012] Map the feature atlas according to the linear layer to obtain a feature vector set;

[0013] Use the fusion calculation layer to fuse the feature vector set to obtain the fused features;

[0014] The fused features are classified and detected according to the linear classifier to obtain the detection results.

[0015] In one embodiment, the data enhancement module is used to enhance the disguised video data to obtain enhanced disguised video data, including:

[0016] In the data enhancement module, the two-dimensional contour points in the disguised video data are orthogonally projected onto the three-dimensional model surface using comparative orthogonal projection to obtain the projection points of the two-dimensional contour points on the three-dimensional object plane.

[0017] The idea of ​​convex hull edge evolution is adopted, continuous curves are used to express edges, a mapping formula is defined, and the projection points are mapped according to the mapping formula to obtain enhanced camouflaged video data.

[0018] In one embodiment, the data enhancement module uses comparative orthogonal projection to perform orthogonal projection on the two-dimensional contour points in the disguised video data and the three-dimensional model surface to obtain the projection points of the two-dimensional contour points on the three-dimensional object plane, including:

[0019] In the data enhancement module, the comparative orthogonal projection is used to orthogonally project the two-dimensional contour points in the disguised video data onto the three-dimensional model surface, and the projection point of the two-dimensional contour point on the three-dimensional object plane is obtained as follows:

[0020]

[0021] Among them, R,t,s are posture parameters, is the rotation matrix, is a two-dimensional translation vector, s is the ratio of the camera focal length to the average distance between the virtual camera and the object, v represents the coordinates of the two-dimensional contour points.

[0022] In one embodiment, mapping the projection points according to a mapping formula to obtain enhanced disguised video data includes:

[0023] The projection points are mapped according to the mapping formula, and the enhanced disguised video data is obtained as follows:

[0024]

[0025] Among them, α represents the shape parameter, (x,y) T Indicates that any pixel point in the two-dimensional image data is taken. Contains S principal components, is the average shape of the three-dimensional object surface composed of three-dimensional points, and B(α, R, t, s) is the three-dimensional face contour.

[0026] In one embodiment, the feature extractor includes a global-branch extractor and a local-branch extractor.

[0027] In one embodiment, extracting features from an image dataset using a feature extractor to obtain a feature map includes:

[0028] Perform large-scale feature extraction on the image dataset using the Global-branch branch extractor to obtain a 64×64 feature map.

[0029] The global-branch branch extractor is used to extract small-size features from the image dataset to obtain a feature map of size 7×7.

[0030] In one embodiment, the novel neural network architecture further includes a multi-head self-attention layer, a normalization layer, a feed-forward linear layer, and a masked self-attention layer.

[0031] In one embodiment, a fusion calculation layer is used to perform feature fusion on a feature vector set to obtain fused features, including:

[0032] The fusion calculation layer is used to fuse the feature vector set, and the fused features are

[0033]

[0034] k=x′ l W k ; v = x′ l W v

[0035]

[0036] CA(x′ l )=Av

[0037]

[0038]

[0039] in, represents the fused features, are all learnable parameters, C and h represent the hidden layer dimension and the number of attention heads of the new neural network architecture, respectively, and f l (·) and g l (·) represent the mapping functions used to transform the two branch vector dimensions into the same, x i Represents the sequence of feature vectors extracted by the convolutional neural network, where i is l or s, representing the sequence corresponding to the large feature map and the small feature map, respectively. represents any eigenvector in the eigenvector set, Indicates the image block vector corresponding to the small feature map, Represents the large feature map class vector, x′ l Represents the vector of the fused features corresponding to the large feature map, q represents the problem in the Transformer architecture, T represents the transpose operation, A represents the fused feature map, and v represents the value in the Transformer architecture. Represents the block vector corresponding to the large feature map, x j Represents the vector corresponding to the feature map, w g,j represents the weight of the mapping function, Both represent the weight of the mapping function, j represents the sequence number of the feature map, LN(·) represents LayerNorm, which is a normalization operation in neural networks, and MCA(·) represents the multi-head fusion attention calculation mechanism.

[0040] A deep fake video detection device based on a novel neural network architecture, comprising:

[0041] The model building module is used to obtain the disguised video data to be detected; build a new neural network structure; the new neural network architecture includes a data enhancement module, an MTCNN algorithm module, a feature extractor, a linear layer, a fusion calculation layer, and a linear classifier;

[0042] A model training module is used to train the new neural network structure according to the pre-acquired training set to obtain a trained new neural network structure;

[0043] A data enhancement module is used to input the disguised video data to be detected into the trained new neural network, and use the data enhancement module to enhance the disguised video data to obtain enhanced disguised video data;

[0044] The recognition module is used to identify the disguised video data according to the MTCNN algorithm module to obtain an image dataset;

[0045] The feature extraction and mapping module is used to extract features from the image dataset according to the feature extractor to obtain a feature atlas; and to map the feature atlas using a linear layer to obtain a feature vector set;

[0046] The feature fusion and classification detection module is used to use the fusion calculation layer to perform feature fusion on the feature vector set to obtain the fused features; and to perform classification detection on the fused features according to the linear classifier to obtain the detection results.

[0047] The aforementioned deepfake video detection method and device based on a novel neural network architecture first constructs a novel neural network structure; the novel neural network architecture includes a data enhancement module, an MTCNN algorithm module, a feature extractor, a linear layer, a fusion computation layer, and a linear classifier. The novel neural network structure is then trained based on a pre-acquired training set to obtain a trained novel neural network structure. The disguised video data to be detected is then input into the trained novel neural network, and the data enhancement module is used to perform data enhancement on the disguised video data to obtain enhanced disguised video data. Data enhancement significantly improves model detection. By strongly enhancing the training data, subsequent detection models can achieve better detection accuracy and generalization capabilities. When the deep fake detection model is deployed to a dataset of unknown origin for detection, data augmentation can improve the model's scalability to achieve better results. The disguised video data is then identified based on the MTCNN algorithm module, which facilitates subsequent feature extraction of the image data. Feature extraction is performed on the image dataset based on the feature extractor, and the feature extractor is divided into two branches. By setting feature extractors of different branches to extract features at different levels, the feature atlas is enriched, so that more accurate features can be extracted after feature mapping and feature fusion. This application performs feature fusion on the feature vector set by setting a fusion calculation layer. The feature vectors in the feature vector set are divided into two branches. First, the feature vector of a branch is used as an agent to interact with all block vectors on the other branch, and then it is put back to the original branch. In this way, the feature vector can fully learn the information on this branch through the self-attention mechanism before the fusion attention calculation is performed. The features extracted by the convolutional neural networks at different levels are extracted and fused, which can enable the model to pay better attention to the global and local features of the image at the same time. Then, by calculating the fusion attention, the final classification basis is made more comprehensive and reliable, thereby improving the accuracy of deep fake video detection. On the one hand, this application can achieve better detection results without using complex methods such as model integration and knowledge distillation. On the other hand, compared with traditional pure convolutional neural networks, it achieves better generalization capabilities and achieves good results in model interpretability. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 11 is a flow chart of a method for detecting deep fake videos based on a novel neural network architecture in one embodiment;

[0049] Figure 2 2 is a structural block diagram of a deep fake video detection device based on a novel neural network architecture in one embodiment. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0051] In one embodiment, Figure 1 As shown, a deep fake video detection method based on a novel neural network architecture is provided, comprising the following steps:

[0052] Step 102, obtain the disguised video data to be detected; construct a new neural network structure; the new neural network architecture includes a data enhancement module, an MTCNN algorithm module, a feature extractor, a linear layer, a fusion calculation layer and a linear classifier.

[0053] Step 104: Train the new neural network structure according to the pre-acquired training set to obtain a trained new neural network structure.

[0054] The new neural network architecture also includes a multi-head self-attention layer, a normalization layer, a feedforward linear layer, and a masked self-attention layer. During network structure training, the multi-head self-attention layer allows the model to pay attention to multiple aspects of information, learning different aspects of the input separately. It also allows parameters to be trained simultaneously, improving model training efficiency. The normalization layer performs residual connections and initialization, aiming to ensure that the model can still be trained and converged even with a large number of deep layers. The masked self-attention layer prevents the calculation of unprocessed input structures when calculating attention parameters, improving model convergence efficiency.

[0055] Step 106: input the disguised video data to be detected into the trained new neural network, and use the data enhancement module to perform data enhancement on the disguised video data to obtain enhanced disguised video data.

[0056] Data augmentation significantly improves model detection performance. By applying strong data augmentation to training data, subsequent detection models can achieve better detection accuracy and generalization capabilities. When deploying deepfake detection models to datasets from unknown sources, data augmentation can improve the model's scalability and achieve better results.

[0057] Step 108: Identify the disguised video data using the MTCNN algorithm module to obtain an image dataset.

[0058] The disguised video data is identified based on the MTCNN algorithm module, which facilitates the subsequent feature extraction of the image data.

[0059] Step 110: extract features from the image dataset using a feature extractor to obtain a feature atlas; and map the feature atlas using a linear layer to obtain a feature vector set.

[0060] The feature extractor is used to extract features from the image dataset. The feature extractor is divided into two branches. One branch is the global branch that extracts a feature map of 64×64 size, and the other branch is the local branch that extracts a feature map of 7×7 size. By setting feature extractors of different branches to extract features at different levels, the feature atlas is enriched, so that more accurate features can be extracted after subsequent feature mapping and feature fusion.

[0061] Step 112: perform feature fusion on the feature vector set using a fusion calculation layer to obtain fused features; perform classification detection on the fused features using a linear classifier to obtain detection results.

[0062] This application performs feature fusion on the feature vector set by setting a fusion calculation layer. The feature vectors in the feature vector set are divided into two branches. First, the feature vector of one branch is used as a proxy to interact with all the block vectors on the other branch, and then it is put back to the original branch. In this way, the feature vector can fully learn the information on the branch through the self-attention mechanism before the fusion attention calculation is performed.

[0063] By extracting and fusing the features extracted by convolutional neural networks at different levels, the model can better focus on the global and local features of the image at the same time. Then, by calculating the fused attention, the final classification basis is made more comprehensive and reliable, thereby improving the accuracy of deep camouflage video detection.

[0064] In the aforementioned deepfake video detection method based on a novel neural network architecture, a novel neural network structure is first constructed. The novel neural network architecture includes a data augmentation module, an MTCNN algorithm module, a feature extractor, a linear layer, a fusion computation layer, and a linear classifier. The novel neural network structure is then trained using a pre-acquired training set to obtain a trained novel neural network structure. The disguised video data to be detected is then input into the trained novel neural network, and the data augmentation module is used to perform data augmentation on the disguised video data to obtain enhanced disguised video data. Data augmentation is particularly effective in improving model detection. By strongly augmenting the training data, subsequent detection models can achieve better detection accuracy and generalization capabilities. When the deep fake detection model is deployed to a dataset of unknown origin for detection, data augmentation can improve the model's scalability to achieve better results. The disguised video data is then identified based on the MTCNN algorithm module, which facilitates subsequent feature extraction of the image data. Feature extraction is performed on the image dataset based on the feature extractor, and the feature extractor is divided into two branches. By setting feature extractors of different branches to extract features at different levels, the feature atlas is enriched, so that more accurate features can be extracted after feature mapping and feature fusion. This application performs feature fusion on the feature vector set by setting a fusion calculation layer. The feature vectors in the feature vector set are divided into two branches. First, the feature vector of a branch is used as an agent to interact with all block vectors on the other branch, and then it is put back to the original branch. In this way, the feature vector can fully learn the information on this branch through the self-attention mechanism before the fusion attention calculation is performed. The features extracted by the convolutional neural networks at different levels are extracted and fused, which can enable the model to pay better attention to the global and local features of the image at the same time. Then, by calculating the fusion attention, the final classification basis is made more comprehensive and reliable, thereby improving the accuracy of deep fake video detection. On the one hand, this application can achieve better detection results without using complex methods such as model integration and knowledge distillation. On the other hand, compared with traditional pure convolutional neural networks, it achieves better generalization capabilities and achieves good results in model interpretability.

[0065] In one embodiment, the data enhancement module is used to enhance the disguised video data to obtain enhanced disguised video data, including:

[0066] In the data enhancement module, the two-dimensional contour points in the disguised video data are orthogonally projected onto the three-dimensional model surface using comparative orthogonal projection to obtain the projection points of the two-dimensional contour points on the three-dimensional object plane.

[0067] The idea of ​​convex hull edge evolution is adopted, continuous curves are used to express edges, a mapping formula is defined, and the projection points are mapped according to the mapping formula to obtain enhanced camouflaged video data.

[0068] In one embodiment, the data enhancement module uses comparative orthogonal projection to perform orthogonal projection on the two-dimensional contour points in the disguised video data and the three-dimensional model surface to obtain the projection points of the two-dimensional contour points on the three-dimensional object plane, including:

[0069] In the data enhancement module, the comparative orthogonal projection is used to orthogonally project the two-dimensional contour points in the disguised video data onto the three-dimensional model surface, and the projection point of the two-dimensional contour point on the three-dimensional object plane is obtained as follows:

[0070]

[0071] Among them, R,t,s are posture parameters, is the rotation matrix, is a two-dimensional translation vector, s is the ratio of the camera focal length to the average distance between the virtual camera and the object, v represents the coordinates of the two-dimensional contour points.

[0072] In one embodiment, mapping the projection points according to a mapping formula to obtain enhanced disguised video data includes:

[0073] The projection points are mapped according to the mapping formula, and the enhanced disguised video data is obtained as follows:

[0074]

[0075] Among them, α represents the shape parameter, (x,y) T Indicates that any pixel point in the two-dimensional image data is taken. Contains S principal components, is the average shape of the three-dimensional object surface composed of three-dimensional points, and B(α, R, t, s) is the three-dimensional face contour.

[0076] In a specific embodiment, data enhancement mainly involves projecting a two-dimensional face video onto a three-dimensional object through a three-dimensional mapping method to obtain a three-dimensional face effect, thereby improving the accuracy of subsequent feature recognition and extraction. First, a proportional orthogonal projection is used to orthogonally project the contour points of the two-dimensional image onto the three-dimensional model surface. The proportional orthogonal function P is used to POS (v, R, t, s), calculate the projection point of the two-dimensional image contour point on the three-dimensional object plane

[0077]

[0078] Among them, R, t, s are posture parameters. is the rotation matrix, is a two-dimensional translation vector, s is the ratio of the camera focal length to the average distance between the virtual camera and the object,

[0079] Expanding on this, the relationship between two-dimensional points and three-dimensional points can be expressed as

[0080] x proj =K[R|t]X 3D

[0081] Where K represents the intrinsic parameter matrix of the virtual camera, [R|t] is the photographic transformation relationship between the camera coordinate system and the world coordinate system,

[0082]

[0083] Then we can get the following equation:

[0084]

[0085] Then, we select four vertices of the two-dimensional image and solve the above equation to get the photographic transformation relationship between the camera coordinate system and the world coordinate system. However, since the three-dimensional object is set up using three-dimensional models such as cups and pillows, it is essentially a deformation network, and its shape is composed of multiple three-dimensional sampling points. Any of its faces can be approximately represented as

[0086]

[0087] Where: Contains S principal components, K is the number of vertices, For the average shape, the vector f(α) contains the coordinates of K vertices, which are combined to form a long vector.

[0088] After mapping the edge points of the two-dimensional image to the surface of the three-dimensional object, the idea of ​​convex hull edge evolution is adopted, the edge is expressed using a continuous curve, and the mapping formula is defined so that its independent variable includes the edge curve. At this time, the mapping process is transformed into a process of solving the minimum value of the mapping amount. The mapping formula can be defined as

[0089]

[0090] Among them E total [v(s)] is the total mapping amount, E ini [v(s)] is the initial contour mapping amount, E image [v(s)] is the mapping amount of the contour on the image, E con [v(s)] is the variance-related term. Through step-by-step iterative optimization, the contour lines of the two-dimensional image can be well mapped onto the surface of the three-dimensional object, ensuring the smoothness of the contour and the topology of the three-dimensional structure when the mapping amount is minimized.

[0091] Then, for the pixel point at the end point of the two-dimensional image, first update the corresponding relationship of the model edge based on the edge point and contour line, so as to obtain the three-dimensional point projection formula of the corresponding relationship, and then calculate the predicted projection position, and calculate the average projection error of the two-dimensional marker point position as

[0092]

[0093] In view of the fact that deep fake detection is basically based on facial images, the algorithm also considers the relative position relationship of facial feature points separately, so as to ensure that the facial image still maintains the original facial features when mapped to the three-dimensional object. For each landmark point of the two-dimensional cheek contour, its corresponding three-dimensional contour vertex is updated to the cheek contour vertex that is closest to it. Using the two-dimensional mapping points and two-dimensional facial landmark points after mapping the three-dimensional facial grid points, the mapping formula is optimized according to the average projection error, and a minimum energy is defined to optimize the facial contour. The enhanced disguised video data is obtained as follows:

[0094]

[0095] Where B(α, R, t, s) is the three-dimensional facial contour.

[0096] For the salient points and contour grid points of the projected facial model, we first find the nearest image points and cheek edge points on the image and find their correspondence. Then, combined with the two-dimensional benchmark landmark points and benchmark cheek points, we update the posture parameters and shape parameters, and then update the correspondence between the model's salient points and facial contours. Finally, we can obtain a facial image mapped onto the surface of a three-dimensional object.

[0097] In one embodiment, the feature extractor includes a global-branch extractor and a local-branch extractor.

[0098] In one embodiment, extracting features from an image dataset using a feature extractor to obtain a feature map includes:

[0099] Perform large-scale feature extraction on the image dataset using the Global-branch branch extractor to obtain a 64×64 feature map.

[0100] The global-branch branch extractor is used to extract small-size features from the image dataset to obtain a feature map of size 7×7.

[0101] In one embodiment, the novel neural network architecture further includes a multi-head self-attention layer, a normalization layer, a feed-forward linear layer, and a masked self-attention layer.

[0102] In one embodiment, a fusion calculation layer is used to perform feature fusion on a feature vector set to obtain fused features, including:

[0103] The fusion calculation layer is used to fuse the feature vector set, and the fused features are

[0104]

[0105] k=x′ l W k ; v = x′ l W v

[0106]

[0107] CA(x′ l )=Av

[0108]

[0109]

[0110] in, represents the fused features, are all learnable parameters, C and h represent the hidden layer dimension and the number of attention heads of the new neural network architecture, respectively, and f l (·) and g l (·) represent the mapping functions used to transform the two branch vector dimensions into the same, x i Represents the sequence of feature vectors extracted by the convolutional neural network, where i is l or s, representing the sequence corresponding to the large feature map and the small feature map, respectively. represents any eigenvector in the eigenvector set, Indicates the image block vector corresponding to the small feature map, Represents the large feature map class vector, x′ l Represents the vector of the fused features corresponding to the large feature map, q represents the problem in the Transformer architecture, T represents the transpose operation, A represents the fused feature map, and v represents the value in the Transformer architecture. Represents the block vector corresponding to the large feature map, x j Represents the vector corresponding to the feature map, w g,j represents the weight of the mapping function, Both represent the weight of the mapping function, j represents the sequence number of the feature map, LN(·) represents LayerNorm, which is a normalization operation in neural networks, and MCA(·) represents the multi-head fusion attention calculation mechanism.

[0111] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0112] In one embodiment, Figure 2 As shown, a deep fake video detection device based on a novel neural network architecture is provided, comprising: a model construction module 202, a model training module 204, a data enhancement module 206, a recognition module 208, a feature extraction and mapping module 210, and a feature fusion and classification detection module 212, wherein:

[0113] The model building module 202 is used to obtain the disguised video data to be detected; build a new neural network structure; the new neural network architecture includes a data enhancement module, an MTCNN algorithm module, a feature extractor, a linear layer, a fusion calculation layer and a linear classifier;

[0114] The model training module 204 is used to train the new neural network structure according to the pre-acquired training set to obtain a trained new neural network structure;

[0115] The data enhancement module 206 is used to input the disguised video data to be detected into the trained new neural network, and use the data enhancement module to enhance the disguised video data to obtain enhanced disguised video data;

[0116] an identification module 208 for identifying disguised video data according to the MTCNN algorithm module to obtain an image dataset;

[0117] The feature extraction and mapping module 210 is used to extract features from the image data set according to the feature extractor to obtain a feature atlas; and to map the feature atlas using a linear layer to obtain a feature vector set;

[0118] The feature fusion and classification detection module 212 is used to perform feature fusion on the feature vector set using the fusion calculation layer to obtain fused features; and perform classification detection on the fused features according to the linear classifier to obtain detection results.

[0119] For the specific definition of the deep fake video detection device based on the new neural network architecture, please refer to the definition of the deep fake video detection method based on the new neural network architecture above, which will not be repeated here. Each module in the above-mentioned deep fake video detection device based on the new neural network architecture can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0120] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0121] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A deep fake video detection method based on a novel neural network architecture, characterized by: The method comprises: Obtaining disguised video data to be detected; Constructing a new neural network structure; the new neural network architecture includes a data enhancement module, an MTCNN algorithm module, a feature extractor, a linear layer, a fusion calculation layer and a linear classifier; Training the novel neural network structure according to a pre-acquired training set to obtain a trained novel neural network structure; Inputting the disguised video data to be detected into the trained novel neural network, and performing data enhancement on the disguised video data using a data enhancement module to obtain enhanced disguised video data; Identify the enhanced disguised video data according to the MTCNN algorithm module to obtain an image dataset; Using the feature extractor to extract features from the image dataset to obtain a feature atlas; Mapping the feature atlas according to the linear layer to obtain a feature vector set; Using the fusion calculation layer to perform feature fusion on the feature vector set to obtain fused features; Performing classification detection on the fused features according to a linear classifier to obtain a detection result; The disguised video data is enhanced by using a data enhancement module to obtain enhanced disguised video data, including: In the data enhancement module, a comparative orthogonal projection is used to perform orthogonal projection on the two-dimensional contour points in the disguised video data and the three-dimensional model surface to obtain the projection points of the two-dimensional contour points on the three-dimensional object plane; The idea of ​​convex hull edge evolution is adopted, continuous curves are used to express edges, a mapping formula is defined, and the projection points are mapped according to the mapping formula to obtain enhanced disguised video data.

2. The method according to claim 1, characterized in that In the data enhancement module, orthogonal projection is performed on the two-dimensional contour points in the disguised video data and the three-dimensional model surface by using comparative orthogonal projection to obtain the projection points of the two-dimensional contour points on the three-dimensional object plane, including: In the data enhancement module, the two-dimensional contour points in the disguised video data are orthogonally projected onto the three-dimensional model surface by using comparative orthogonal projection, and the projection points of the two-dimensional contour points on the three-dimensional object plane are obtained as follows: Among them, R,t,s are posture parameters, is the rotation matrix, is a two-dimensional translation vector, s is the ratio of the camera focal length to the average distance between the virtual camera and the object, v represents the coordinates of the two-dimensional contour points.

3. The method according to claim 2, characterized in that Mapping the projection points according to a mapping formula to obtain enhanced disguised video data includes: The projection points are mapped according to the mapping formula to obtain the enhanced disguised video data: Among them, α represents the shape parameter, (x,y) T Indicates that any pixel point in the two-dimensional image data is taken. Contains S principal components, is the average shape of the three-dimensional object surface composed of three-dimensional points, and B(α, R, t, s) is the three-dimensional face contour.

4. The method according to claim 1, wherein The feature extractor includes a global-branch extractor and a local-branch extractor.

5. The method according to claim 4, characterized in that Extracting features from the image dataset using the feature extractor to obtain a feature map includes: Performing large-scale feature extraction on the image dataset according to the Global-branch branch extractor to obtain a feature map of size 64×64; The image dataset is subjected to small-size feature extraction according to the Global-branch branch extractor to obtain a feature map of size 7×7.

6. The method according to any one of claims 1 to 5, characterized in that The novel neural network architecture also includes a multi-head self-attention layer, a normalization layer, a feed-forward linear layer, and a masked self-attention layer.

7. The method according to claim 1, characterized in that The feature vector set is subjected to feature fusion by using the fusion calculation layer to obtain fused features, including: The fusion calculation layer is used to perform feature fusion on the feature vector set, and the fused feature is obtained as follows: CA(x ′l )=Off in, represents the fused features, W q ,W k , are all learnable parameters, C and h represent the hidden layer dimension and the number of attention heads of the new neural network architecture, respectively, and f l (·) and g l (·) represent the mapping functions used to transform the two branch vector dimensions into the same, x i Represents the sequence of feature vectors extracted by the convolutional neural network, where i is l or s, representing the sequence corresponding to the large feature map and the small feature map, respectively. represents any eigenvector in the eigenvector set, Indicates the image block vector corresponding to the small feature map, Represents a large feature map class vector, x ′l Represents the vector of the fused features corresponding to the large feature map, q represents the problem in the Transformer architecture, T represents the transpose operation, A represents the fused feature map, and v represents the value in the Transformer architecture. Represents the block vector corresponding to the large feature map, x j Represents the vector corresponding to the feature map, w g,j represents the weight of the mapping function, Both represent the weight of the mapping function, j represents the sequence number of the feature map, LN(·) represents LayerNorm, which is a normalization operation in neural networks, and MCA(·) represents the multi-head fusion attention calculation mechanism.

8. A deep fake video detection device based on a novel neural network architecture, characterized in that: The device comprises: A model building module is used to obtain disguised video data to be detected; and to build a new neural network structure; the new neural network architecture includes a data enhancement module, an MTCNN algorithm module, a feature extractor, a linear layer, a fusion calculation layer, and a linear classifier; A model training module is used to train the novel neural network structure according to a pre-acquired training set to obtain a trained novel neural network structure; A data enhancement module is used to input the disguised video data to be detected into the trained novel neural network, and perform data enhancement on the disguised video data using the data enhancement module to obtain enhanced disguised video data, including: In the data enhancement module, a comparative orthogonal projection is used to perform orthogonal projection on the two-dimensional contour points in the disguised video data and the three-dimensional model surface to obtain the projection points of the two-dimensional contour points on the three-dimensional object plane; Adopting the idea of ​​convex hull edge evolution, using continuous curves to express edges, defining a mapping formula, and mapping the projection points according to the mapping formula to obtain enhanced disguised video data; an identification module, configured to identify the disguised video data according to the MTCNN algorithm module to obtain an image data set; A feature extraction and mapping module, configured to extract features from the image dataset using the feature extractor to obtain a feature atlas; and map the feature atlas using the linear layer to obtain a feature vector set; The feature fusion and classification detection module is used to use the fusion calculation layer to perform feature fusion on the feature vector set to obtain fused features; and to perform classification detection on the fused features according to a linear classifier to obtain detection results.

Images