ResNet and LSTM (Long Short Term Memory)-based multi-dimensional spatio-temporal feature fusion deep counterfeit video detection method and device

Abstract

The invention discloses a ResNet and LSTM-based multi-dimensional spatial-temporal feature fusion deep counterfeit video detection method and device, and the method comprises the steps: obtaining a face image in a video, and positioning key points in the face image; extracting a spatial feature vector in the face image based on a pre-trained ResNet model; performing feature enhancement on the spatial feature vector extracted by the ResNet model based on an attention mechanism; inputting the enhanced spatial feature vector extracted from each frame of face image into an LSTM network according to a time sequence to obtain a time sequence feature; and fusing the time sequence features output by the LSTM and the spatial features extracted by the ResNet model to obtain a fused feature vector, inputting the fused feature vector into a full connection layer for classification, inputting a classification result into a SoftMax layer to calculate the probability of each category, and obtaining the classification probabilities of forged and real videos. According to the invention, through fusion of the multi-dimensional spatial-temporal features, the face counterfeit content in the deep counterfeit video is effectively detected, and the accuracy and robustness of detection are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning, and specifically relates to a deep fake video detection method and device based on ResNet and LSTM with multi-dimensional spatiotemporal feature fusion. Background Art

[0002] In recent years, with the rapid progress in the field of deep learning, advanced algorithms such as generative adversarial networks (GANs) have made remarkable achievements in creating highly realistic synthetic content, especially in deepfake technology, which can simulate and reconstruct facial expressions, movements and even voices with high fidelity, making the synthetic video almost indistinguishable from the real video in visual effects. The abuse of this technology has become a serious social problem because it is used to spread false information, cause political turmoil and infringe on personal privacy, thus posing a serious challenge to public safety.

[0003] At present, face recognition detection technologies for deep fakes are mainly divided into three categories: image feature-based methods, time series information-based methods, and multimodal fusion methods. Image feature-based technologies usually rely on convolutional neural networks (CNNs) to extract spatial features in static images and identify signs of forgery by analyzing facial details such as texture, lighting, and morphology. Time series information-based technologies use recurrent neural networks (RNNs) or long short-term memory networks (LSTMs) to analyze the dynamic relationship between video frames and detect unnatural expression changes or temporal incoherence. Multimodal fusion methods, on the other hand, make comprehensive use of multiple information sources such as vision and hearing to enhance the accuracy and reliability of detection.

[0004] Although these methods have improved the detection capabilities of deep fake videos to a certain extent, they still face some key challenges. For example, methods based on image features tend to focus on spatial information within a single frame and ignore the temporal correlation between frames, which may not effectively identify videos with no obvious signs of tampering in a single frame but with timing anomalies. Although methods based on time series information can capture inconsistencies in the temporal dimension, their detection performance is still insufficient for videos with long duration or containing complex scenes. At the same time, although multimodal fusion methods can integrate multiple types of information, they have high computational costs and limited adaptability to emerging forgery technologies. Therefore, there is an urgent need to develop a new detection technology that can comprehensively consider multi-dimensional spatiotemporal features and has high robustness. Summary of the invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a deep fake video detection method and device based on ResNet and LSTM with multi-dimensional spatiotemporal feature fusion. The present invention combines the powerful spatial feature extraction capability of convolutional neural network (ResNet) with the time series analysis capability of long short-term memory network (LSTM), aiming to effectively detect the fake face content in deep fake videos through the fusion of multi-dimensional spatiotemporal features, thereby improving the accuracy and robustness of detection.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a deep fake video detection method based on multi-dimensional spatiotemporal feature fusion of ResNet and LSTM, comprising the following steps:

[0008] Get the face image in the video and locate the key points in the face image;

[0009] Extracting spatial feature vectors in the face image based on a pre-trained ResNet model, wherein the spatial feature vectors contain important spatial information in the face image;

[0010] Based on the attention mechanism, the spatial feature vector extracted by the ResNet model is enhanced. The attention weight of each time step is calculated through the attention mechanism, and the attention weight is applied to the original spatial feature vector to obtain the enhanced spatial feature vector.

[0011] The enhanced spatial feature vector extracted from each frame of the face image is input into the LSTM network in time order to obtain time series features; the LSTM network captures the time dependency between video frames through its memory unit and gating mechanism, and analyzes the time series information in the video;

[0012] The time series features output by the LSTM network are fused with the spatial features extracted by the ResNet model to obtain a fused feature vector, which is input into the fully connected layer for classification. The classification result is input into the SoftMax layer to calculate the probability of each category, and the classification probability of the forged and real videos is obtained. The fused feature vector contains the temporal and spatial information in the video and can fully reflect the changes in the face in the video.

[0013] As a preferred technical solution, the method of acquiring a face image in a video and locating key points in the face image is specifically as follows:

[0014] Through a preset multi-task cascade convolutional network, the face area in the video frame is located, and bounding box regression and key point positioning are performed, and the detected face area is cropped and standardized to a fixed size; the key points include two eyes, the tip of the nose and two corners of the mouth.

[0015] As a preferred technical solution, the spatial feature vector in the face image is extracted based on the pre-trained ResNet model, specifically:

[0016] Input the cropped and normalized face image into the pre-trained ResNet model;

[0017] The ResNet model extracts high-dimensional spatial features of face images through convolutional layers and residual blocks.

[0018] As a preferred technical solution, the feature enhancement of the spatial feature vector extracted by the ResNet model based on the attention mechanism is specifically as follows:

[0019] The spatial feature vector extracted by the ResNet model is input into the adaptive feature enhancement module;

[0020] For the attention mechanism in the adaptive feature enhancement module, the attention weight of each time step is calculated. Specifically, the similarity between the feature vector and the attention vector is calculated based on the hidden state of the current time step, the attention state of the previous time step, the trainable attention vector and the trainable weight matrix; then the attention weight is calculated based on the similarity and the softmax function; then the attention weight is applied to the original feature vector to generate an enhanced feature vector; finally, the enhanced feature vector will be used as the feature of the LSTM network input to further improve the accuracy of time series feature analysis.

[0021] As a preferred technical solution, the LSTM network includes:

[0022] An input gate, which determines the importance of the current input information and controls the proportion of the information entering the memory unit;

[0023] A forget gate, which determines how much memory from the previous time step is retained in the current time step;

[0024] An output gate, which determines how the state of the memory cell at the current time step will affect the hidden state at the next time step;

[0025] Memory cells, which determine how the network's long-term memory is updated at each time step;

[0026] The hidden state reflects the output of the current time step and is determined by the state of the memory unit and the output gate.

[0027] As a preferred technical solution, the time series features output by the LSTM network are fused with the spatial features extracted by the ResNet model to obtain a fused feature vector, specifically:

[0028] The final LSTM network hidden state H t ResNet feature vector after enhancement module Fusion to form the final feature vector Among them, Concat(.) represents the concatenation operation of vectors.

[0029] As a preferred technical solution, the fused feature vector is input into the fully connected layer for classification, specifically:

[0030] The fused features are processed using a linear layer that accepts 2048-dimensional input features and outputs 2 features for binary classification tasks;

[0031] Use Leaky ReLU activation function to enhance the nonlinear expression ability of the model;

[0032] The features are mapped to the target output size through an adaptive average pooling layer, and the confidence of the model in the fake and real classification is calculated through a SoftMax layer;

[0033] The output of the SoftMax layer is used to determine whether the input video is a deep fake.

[0034] In the second aspect, the present invention provides a deep fake video detection system based on ResNet and LSTM with multi-dimensional spatiotemporal feature fusion, which is applied to the deep fake video detection method based on ResNet and LSTM with multi-dimensional spatiotemporal feature fusion, including a face image acquisition module, a spatial feature extraction module, a feature enhancement module, a temporal feature analysis module, and a feature fusion and classification module;

[0035] The face image acquisition module is used to acquire the face image in the video and locate the key points in the face image;

[0036] The spatial feature extraction module is used to extract spatial feature vectors in the face image based on the pre-trained ResNet model, and the spatial feature vectors contain important spatial information in the face image;

[0037] The feature enhancement module is used to perform feature enhancement on the spatial feature vector extracted by the ResNet model based on the attention mechanism, calculate the attention weight of each time step through the attention mechanism, and apply the attention weight to the original spatial feature vector, so as to obtain the enhanced spatial feature vector;

[0038] The temporal feature analysis module is used to input the enhanced spatial feature vector extracted from each frame of the face image into the LSTM network in chronological order to obtain the temporal series feature; the LSTM network captures the temporal dependency between video frames through its memory unit and gating mechanism, and analyzes the temporal information in the video;

[0039] The feature fusion and classification module is used to fuse the time series features output by the LSTM network with the spatial features extracted by the ResNet model to obtain a fused feature vector, input the fused feature vector into the fully connected layer for classification, and input the classification result into the SoftMax layer to calculate the probability of each category to obtain the classification probability of fake and real videos; the fused feature vector contains the temporal and spatial information in the video and can fully reflect the changes in the face in the video.

[0040] In a third aspect, the present invention provides an electronic device, the electronic device comprising:

[0041] at least one processor; and,

[0042] a memory communicatively connected to the at least one processor; wherein,

[0043] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute a deep fake video detection method based on multi-dimensional spatiotemporal feature fusion of ResNet and LSTM.

[0044] In a fourth aspect, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the multi-dimensional spatiotemporal feature fusion deep fake video detection method based on ResNet and LSTM.

[0045] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0046] 1. Multi-dimensional spatiotemporal feature fusion to improve detection accuracy: The present invention combines the spatial feature extraction capability of ResNet and the time series analysis capability of LSTM, and can fully capture the forgery traces in the video by fusing multi-dimensional spatiotemporal features. Compared with detection methods that rely only on spatial or temporal features, the comprehensive feature vector of the present invention can more accurately identify subtle forgeries in the video, significantly improving the accuracy of detection.

[0047] 2. LSTM timing analysis accurately captures time dependencies in videos: LSTM networks can capture unnatural time series changes in videos, such as sudden changes in facial expressions or discontinuities in motion, by processing the time dependencies between consecutive frames. Compared with traditional static image detection methods, this invention can more effectively identify abnormal changes in long-term forged videos through LSTM timing processing.

[0048] 3. Adaptive feature enhancement to improve the expressiveness of key features: By introducing an adaptive feature enhancement module, the present invention can dynamically adjust the weight of features according to the contextual information of the video frame to enhance the expressiveness of key features. This method can help the model more accurately capture areas where there may be traces of forgery and improve the overall detection effect, especially in complex or well-disguised forged videos.

[0049] 4. Efficient classification and probability prediction to improve detection reliability: By fusing spatial and temporal feature vectors and using fully connected layers for classification, the present invention can provide accurate forged video detection results. In addition, the classification probability output by the SoftMax layer can quantify the uncertainty of the detection results, providing a more reliable basis for decision-making in application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 This is a flow chart of a deep fake video detection method based on multi-dimensional spatiotemporal feature fusion of ResNet and LSTM in an embodiment of the present invention;

[0052] Figure 2 This is a block diagram of a deep fake video detection system based on multi-dimensional spatiotemporal feature fusion of ResNet and LSTM according to an embodiment of the present invention.

[0053] Figure 3 4 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0055] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0056] like Figure 1 As shown, this embodiment provides a deep fake video detection method based on multi-dimensional spatiotemporal feature fusion of ResNet and LSTM, including the following steps:

[0057] S1. Obtain the face image in the video and locate the key points in the face image.

[0058] Specifically, in the input video frame, face detection is first performed on each frame of the image through the MTCNN (Multi-task Cascaded Convolutional Networks) algorithm. MTCNN accurately locates the face area in the video frame through its cascaded convolutional network structure, and regresses the bounding box and locates the five key points (eyes, nose tip, and mouth corners). This step ensures that we only process the face part of the video, rather than the entire scene, thereby improving the efficiency and accuracy of subsequent feature extraction. The detected face area will be cropped and standardized to a fixed size (for example, 112x112 pixels) so that the subsequent ResNet model can process it uniformly. This step can reduce background interference and focus on the detection of forgery traces in the face area.

[0059] S2. Extracting spatial feature vectors in the face image based on the pre-trained ResNet model, wherein the spatial feature vectors contain important spatial information in the face image.

[0060] Specifically, a pre-trained ResNet model is used for feature extraction, and the cropped and standardized face image is then input into a pre-trained ResNet model. ResNet, with its residual network structure, can effectively extract high-level spatial features of the image. In the present invention, we used the ResNet-50 model, which contains a 50-layer deep network that can capture detailed features in face images, such as texture, lighting, facial structure, etc. These features are crucial for detecting subtle traces of forgery in deep forgeries. After the last pooling layer of ResNet, a 2048-dimensional feature vector is generated. These feature vectors represent the representation of the input image in a high-dimensional space and contain rich spatial information.

[0061] S3. Based on the attention mechanism, the spatial feature vector extracted by the ResNet model is enhanced. The attention weight of each time step is calculated through the attention mechanism, and the attention weight is applied to the original spatial feature vector to obtain the enhanced spatial feature vector.

[0062] Specifically, the spatial feature vector extracted by the ResNet model is input into the adaptive feature enhancement module;

[0063] For the attention mechanism in the adaptive feature enhancement module, the attention weight of each time step is calculated as follows:

[0064] S31, firstly, based on the hidden state of the current time step, the attention state of the previous time step, the trainable attention vector and the trainable weight matrix, calculate the similarity between the feature vector and the attention vector;

[0065] S32, then calculate the attention weight based on the similarity and the softmax function; then apply the attention weight to the original feature vector to generate an enhanced feature vector;

[0066] S33. Finally, the enhanced feature vector will be used as the input feature of the LSTM network to further improve the accuracy of time series feature analysis.

[0067] S4. The enhanced spatial feature vector extracted from each frame of the face image is input into the LSTM network in chronological order to obtain time series features; the LSTM network captures the time dependency between video frames through its memory unit and gating mechanism, and analyzes the timing information in the video.

[0068] Specifically, the 2048-dimensional feature vector extracted from each frame of the video image is input into the LSTM network in chronological order. Through its memory unit and gating mechanism, LSTM can effectively capture the temporal dependency between video frames and analyze the timing information in the video. Specifically, LSTM detects discontinuous time series changes in the video, such as abnormal changes in facial expressions, by comparing the frames at time points 't' seconds and 't-n' seconds to determine whether there are signs of forgery. We use an LSTM layer with a 2048-dimensional hidden layer and set a Dropout probability of 0.4 to prevent overfitting. The output of the LSTM layer will be used to further analyze the timing characteristics of the video.

[0069] The key components of the LSTM network are as follows:

[0070] A. The input gate determines the importance of the current input information and controls the proportion of it entering the memory unit. The calculation formula of the input gate is:

[0071] i t =σ(W i ·x t +U i ·h t-1 +b i )

[0072] Among them, x t represents the input of the current time step, h t-1 is the hidden state of the previous time step, W i and U i is the weight matrix, b i is the bias term, and σ is the sigmoid activation function.

[0073] B. The forget gate determines how much memory from the previous time step is retained in the current time step. The calculation formula of the forget gate is:

[0074] f t =σ(W f ·x t +U f ·h t-1 +b f )

[0075] Similarly, f t Controls the previous memory cell state c t-1 Impact on current status.

[0076] C. The output gate determines how the memory cell state at the current time step will affect the hidden state at the next time step.

[0077] The calculation formula for going out is:

[0078] ot =σ(W o ·x t +U o ·h t-1 +b o )

[0079] The function of the output gate is to adjust the output ratio according to the current input and the previous hidden state.

[0080] D. Cell State Update is the core function of LSTM, which determines the

[0081] How to update the long-term memory of the network. The update formula of the memory unit is:

[0082] c t =f t ⊙c t-1 +i t ⊙tanh(W c ·x t +U c ·h t-1 +b c )

[0083] Among them, ⊙ represents the element-level multiplication operation, and tanh is the hyperbolic tangent activation function. Through this formula, LSTM can selectively remember or forget information.

[0084] D. The hidden state reflects the output of the current time step and is determined by the state of the memory unit and the output gate.

[0085] The calculation formula is:

[0086] h t =o t ⊙tan c t

[0087] Through these gating mechanisms, the LSTM network can effectively maintain a stable learning effect when processing long-term series data, avoiding the gradient vanishing and explosion problems common in traditional RNNs. This makes LSTM particularly suitable for tasks that require long-term dependencies, such as time series prediction, so that the present invention can detect deep fakes from the image level to the video level.

[0088] S5. The time series features output by the LSTM network are fused with the spatial features extracted by the ResNet model to obtain a fused feature vector. The fused feature vector is input into the fully connected layer for classification. The classification result is input into the SoftMax layer to calculate the probability of each category, and the classification probability of the forged and real videos is obtained. The fused feature vector contains the temporal and spatial information in the video and can fully reflect the changes in the face in the video.

[0089] Specifically, the time series features output by LSTM are fused with the spatial features extracted by ResNet to form a comprehensive feature vector. This feature vector contains the temporal and spatial information in the video and can fully reflect the changes in the face in the video. The fused features are processed using a linear layer, which accepts 2048-dimensional input features and outputs 2 features for binary classification tasks. We also use the Leaky ReLU activation function to enhance the nonlinear expression ability of the model. Finally, the features are mapped to the target output size through an adaptive average pooling layer, and the confidence of the model in the forged and real classification is calculated through a SoftMax layer. The output of the SoftMax layer is used to determine whether the input video is a deep fake.

[0090] In another embodiment of the present invention, a deep fake video detection method based on multi-dimensional spatiotemporal feature fusion of ResNet and LSTM is provided, comprising the following steps:

[0091] S1. Video frame preprocessing and face detection, specifically including the following steps:

[0092] S1.1. Video frame extraction: Extract frame images from the input video at a fixed frame rate, denoted as {It}, where It represents the frame image at time t.

[0093] S1.2, face detection: Use the MTCNN algorithm to perform face detection on each frame image and obtain the face bounding box B t ={X t ,Y t ,W t ,H t} and five key point coordinate sets K t ={(X1,Y1),(X2,Y2),…,(X5,Y5)}, where X t and Y t are the coordinates of the upper left corner of the bounding box, W t and H t for width and height.

[0094] S1.3, face area cropping: according to the detected bounding box B t, the face area is cropped and standardized to a fixed size S×S (such as 112x112 pixels), denoted as

[0095] S2, spatial feature extraction, specifically including the following steps:

[0096] S2.1. Input face image: The cropped face image Input the pre-trained ResNet model.

[0097] S2.2, ResNet feature extraction: The ResNet model extracts high-dimensional spatial features of the image through convolutional layers and residual blocks. The feature vector extracted by ResNet is expressed as:

[0098]

[0099] Among them, F t is a 2048-dimensional feature vector, f ResNet (.) represents the feature extraction function of the ResNet model.

[0100] S2.3. Feature vector generation: The generated feature vector F t It contains important spatial information in the face image and is the key data for subsequent time series analysis.

[0101] The ResNet convolution operation can be expressed as (H is the height, W is the width, and C is the number of channels):

[0102]

[0103] Where K represents the convolution kernel (size is M×N×C×K, K is the number of channels), b is the bias term, and F is the output feature map.

[0104] S3, adaptive feature enhancement, specifically includes the following steps:

[0105] In order to further improve the detection capability of the model, the present invention introduces an adaptive feature enhancement module, which uses the attention mechanism to enhance the feature vector extracted by ResNet. The specific steps are as follows:

[0106] S3.1. Feature vector input: The 2048-dimensional feature vector F extracted by ResNet t Input into the adaptive feature enhancement module.

[0107] S3.2. For the attention mechanism in the adaptive feature enhancement module, the attention weight of each time step can be calculated by the following formula. First, the similarity between the feature vector and the attention vector is calculated:

[0108]

[0109] Among them, h t represents the hidden state of the current time step, S t-1 represents the attention state of the previous time step, v is the trainable attention vector, and W h and W s is the trainable weight matrix, b e is the bias term.

[0110] S3.3, then calculate the attention weight through the softmax function:

[0111]

[0112] S3.4. Feature enhancement: The attention weight α t Applied to the original feature vector to generate an enhanced feature vector The calculation formula is as follows:

[0113]

[0114] S3.5. Output enhanced features: enhanced feature vector The features used as LSTM input can further improve the accuracy of time series feature analysis.

[0115] S4, timing feature analysis, including the following steps:

[0116] S4.1. Input to LSTM network: extract the feature vector of each frame Input the LSTM network in chronological order to form a feature sequence {F t}, expressed as:

[0117]

[0118] Where T is the total number of video frames.

[0119] S4.2. LSTM time series processing: LSTM processes the input feature sequence The time dependency is captured by the memory unit and the gating mechanism, and the sequence feature vector {H t},in:

[0120]

[0121] H t is the hidden state at time t, f LSTM (.) is the mapping function of LSTM, which includes the calculation of input gate, forget gate and output gate.

[0122] S4.3. Output of the temporal feature lSTM network Ht Represents the feature vector after time series processing, which is used for subsequent feature fusion.

[0123] S5, feature fusion and classification, specifically including the following steps:

[0124] S5.1. Feature fusion: The last LSTM hidden state H t ResNet feature vector after enhancement module Fusion to form the final feature vector Among them, Concat(.) represents the concatenation operation of vectors.

[0125] S5.2. Classification processing: fused feature vector F fusion Input to the fully connected layer for classification. Assuming the weight matrix of the fully connected layer is W and the bias vector is b, the output y is expressed as:

[0126] y=W·F fusion +b

[0127] Among them, y is the original output of the classification result.

[0128] S5.3. Probability prediction and SoftMax layer: Input the classification results into the SoftMax layer to calculate the probability of each category and obtain the classification probability P of the fake and real videos fake P real :

[0129] Here, i represents the category index of the classification, and n is the number of categories (in the present invention, n=2).

[0130] S5.4. Result output: According to the probability output by the SoftMax layer, the category with the highest probability is selected as the final judgment result.

[0131] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0132] Based on the same idea as the multi-dimensional spatiotemporal feature fusion deep fake video detection method based on ResNet and LSTM in the above-mentioned embodiment, the present invention also provides a multi-dimensional spatiotemporal feature fusion deep fake video detection system based on ResNet and LSTM, which can be used to execute the above-mentioned multi-dimensional spatiotemporal feature fusion deep fake video detection method based on ResNet and LSTM. For ease of explanation, in the structural schematic diagram of the embodiment of the multi-dimensional spatiotemporal feature fusion deep fake video detection system based on ResNet and LSTM, only the parts related to the embodiment of the present invention are shown. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than the illustrated structure, or combine certain components, or arrange the components differently.

[0133] See also Figure 2 In another embodiment of the present application, a multi-dimensional spatiotemporal feature fusion deep fake video detection system 100 based on ResNet and LSTM is provided, the system comprising a face image acquisition module 101, a spatial feature extraction module 102, a feature enhancement module 103, a temporal feature analysis module 104, and a feature fusion and classification module 105;

[0134] The face image acquisition module 101 is used to acquire the face image in the video and locate the key points in the face image;

[0135] The spatial feature extraction module 102 is used to extract the spatial feature vector in the face image based on the pre-trained ResNet model, and the spatial feature vector contains important spatial information in the face image;

[0136] The feature enhancement module 103 is used to perform feature enhancement on the spatial feature vector extracted by the ResNet model based on the attention mechanism, calculate the attention weight of each time step through the attention mechanism, and apply the attention weight to the original spatial feature vector, so as to obtain the enhanced spatial feature vector;

[0137] The temporal feature analysis module 104 is used to input the enhanced spatial feature vector extracted from each frame of the face image into the LSTM network in time sequence to obtain time series features; the LSTM network captures the time dependency between video frames through its memory unit and gating mechanism, and analyzes the temporal information in the video;

[0138] The feature fusion and classification module 105 is used to fuse the time series features output by the LSTM network with the spatial features extracted by the ResNet model to obtain a fused feature vector, input the fused feature vector into the fully connected layer for classification, and input the classification result into the SoftMax layer to calculate the probability of each category to obtain the classification probability of the forged and real videos; the fused feature vector contains the temporal and spatial information in the video and can fully reflect the changes in the face of the person in the video.

[0139] It should be noted that the multi-dimensional spatiotemporal feature fusion deep fake video detection system based on ResNet and LSTM of the present invention corresponds one to one with the multi-dimensional spatiotemporal feature fusion deep fake video detection method based on ResNet and LSTM of the present invention. The technical features and beneficial effects described in the embodiment of the multi-dimensional spatiotemporal feature fusion deep fake video detection method based on ResNet and LSTM are applicable to the embodiment of multi-dimensional spatiotemporal feature fusion deep fake video detection based on ResNet and LSTM. For specific contents, please refer to the description in the embodiment of the method of the present invention, which will not be repeated here.

[0140] In addition, in the implementation of the implementation of the multi-dimensional spatiotemporal feature fusion deep fake video detection system based on ResNet and LSTM in the above-mentioned embodiment, the logical division of each program module is only an example. In actual applications, the above-mentioned functions can be assigned to different program modules as needed, for example, for the configuration requirements of the corresponding hardware or the convenience of software implementation. That is, the internal structure of the multi-dimensional spatiotemporal feature fusion deep fake video detection system based on ResNet and LSTM is divided into different program modules to complete all or part of the functions described above.

[0141] See also Figure 3 In one embodiment, an electronic device that implements a deep fake video detection method based on multi-dimensional spatiotemporal feature fusion of ResNet and LSTM is provided. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a deep fake video detection program 203 based on multi-dimensional spatiotemporal feature fusion of ResNet and LSTM.

[0142] Among them, the first memory 202 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 200. Further, the first memory 202 can also include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 can not only be used to store application software and various types of data installed in the electronic device 200, such as the code of the deep fake video detection program 203 based on the multi-dimensional spatiotemporal feature fusion of ResNet and LSTM, but also can be used to temporarily store data that has been output or is to be output.

[0143] In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The first processor 201 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes various functions and processes data of the electronic device 200 by running or executing programs or modules stored in the first memory 202, and calling data stored in the first memory 202.

[0144] Figure 3 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on the electronic device 200 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0145] The multi-dimensional spatiotemporal feature fusion deep fake video detection program 203 based on ResNet and LSTM stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions. When running in the first processor 201, it can achieve:

[0146] Get the face image in the video and locate the key points in the face image;

[0147] Extracting spatial feature vectors in the face image based on a pre-trained ResNet model, wherein the spatial feature vectors contain important spatial information in the face image;

[0148] Based on the attention mechanism, the spatial feature vector extracted by the ResNet model is enhanced. The attention weight of each time step is calculated through the attention mechanism, and the attention weight is applied to the original spatial feature vector to obtain the enhanced spatial feature vector.

[0149] The enhanced spatial feature vector extracted from each frame of the face image is input into the LSTM network in time order to obtain time series features; the LSTM network captures the time dependency between video frames through its memory unit and gating mechanism, and analyzes the time series information in the video;

[0150] The time series features output by the LSTM network are fused with the spatial features extracted by the ResNet model to obtain a fused feature vector, which is input into the fully connected layer for classification. The classification result is input into the SoftMax layer to calculate the probability of each category, and the classification probability of the forged and real videos is obtained. The fused feature vector contains the temporal and spatial information in the video and can fully reflect the changes in the face in the video.

[0151] Furthermore, if the module / unit integrated in the electronic device 200 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0152] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0153] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A deep fake video detection method based on multi-dimensional spatiotemporal feature fusion of ResNet and LSTM, characterized by: The steps include: Get the face image in the video and locate the key points in the face image; Extracting spatial feature vectors in the face image based on a pre-trained ResNet model, wherein the spatial feature vectors contain important spatial information in the face image; Based on the attention mechanism, the spatial feature vector extracted by the ResNet model is enhanced. The attention weight of each time step is calculated through the attention mechanism, and the attention weight is applied to the original spatial feature vector to obtain the enhanced spatial feature vector. The enhanced spatial feature vector extracted from each frame of the face image is input into the LSTM network in time order to obtain time series features; the LSTM network captures the time dependency between video frames through its memory unit and gating mechanism, and analyzes the time series information in the video; The time series features output by the LSTM network are fused with the spatial features extracted by the ResNet model to obtain a fused feature vector, which is input into the fully connected layer for classification. The classification result is input into the SoftMax layer to calculate the probability of each category, and the classification probability of the forged and real videos is obtained. The fused feature vector contains the temporal and spatial information in the video and can fully reflect the changes in the face in the video.

2. According to claim 1, the multi-dimensional spatiotemporal feature fusion deep fake video detection method based on ResNet and LSTM is characterized in that: The method of obtaining a face image in a video and locating key points in the face image is as follows: Through a preset multi-task cascade convolutional network, the face area in the video frame is located, and bounding box regression and key point positioning are performed, and the detected face area is cropped and standardized to a fixed size; the key points include two eyes, the tip of the nose and two corners of the mouth.

3. According to claim 1, the multi-dimensional spatiotemporal feature fusion deep fake video detection method based on ResNet and LSTM is characterized in that: The spatial feature vector in the face image is extracted based on the pre-trained ResNet model, specifically: Input the cropped and normalized face image into the pre-trained ResNet model; The ResNet model extracts high-dimensional spatial features of face images through convolutional layers and residual blocks.

4. According to claim 1, the multi-dimensional spatiotemporal feature fusion deep fake video detection method based on ResNet and LSTM is characterized in that: The feature enhancement of the spatial feature vector extracted by the ResNet model based on the attention mechanism is specifically as follows: The spatial feature vector extracted by the ResNet model is input into the adaptive feature enhancement module; For the attention mechanism in the adaptive feature enhancement module, the attention weight of each time step is calculated. Specifically, the similarity between the feature vector and the attention vector is calculated based on the hidden state of the current time step, the attention state of the previous time step, the trainable attention vector and the trainable weight matrix; then the attention weight is calculated based on the similarity and the softmax function; then the attention weight is applied to the original feature vector to generate an enhanced feature vector; finally, the enhanced feature vector will be used as the feature of the LSTM network input to further improve the accuracy of time series feature analysis.

5. According to claim 1, the multi-dimensional spatiotemporal feature fusion deep fake video detection method based on ResNet and LSTM is characterized in that: The LSTM network includes: An input gate, which determines the importance of the current input information and controls the proportion of the information entering the memory unit; A forget gate, which determines how much memory from the previous time step is retained in the current time step; An output gate, which determines how the state of the memory cell at the current time step will affect the hidden state at the next time step; Memory cells, which determine how the network's long-term memory is updated at each time step; The hidden state reflects the output of the current time step and is determined by the state of the memory unit and the output gate.

6. According to claim 1, the multi-dimensional spatiotemporal feature fusion deep fake video detection method based on ResNet and LSTM is characterized in that: The time series features output by the LSTM network are fused with the spatial features extracted by the ResNet model to obtain a fused feature vector, specifically: The final LSTM network hidden state H t ResNet feature vector after enhancement module Fusion to form the final feature vector Among them, Concat(.) represents the concatenation operation of vectors.

7. According to claim 1, the multi-dimensional spatiotemporal feature fusion deep fake video detection method based on ResNet and LSTM is characterized in that: The fused feature vector is input into the fully connected layer for classification, specifically: The fused features are processed using a linear layer that accepts 2048-dimensional input features and outputs 2 features for binary classification tasks; Use Leaky ReLU activation function to enhance the nonlinear expression ability of the model; The features are mapped to the target output size through an adaptive average pooling layer, and the confidence of the model in the fake and real classification is calculated through a SoftMax layer; The output of the SoftMax layer is used to determine whether the input video is a deep fake.

8. A deep fake video detection system based on multi-dimensional spatiotemporal feature fusion of ResNet and LSTM, characterized by: The multi-dimensional spatiotemporal feature fusion deep fake video detection method based on ResNet and LSTM applied to any one of claims 1-7 comprises a face image acquisition module, a spatial feature extraction module, a feature enhancement module, a temporal feature analysis module, and a feature fusion and classification module; The face image acquisition module is used to acquire the face image in the video and locate the key points in the face image; The spatial feature extraction module is used to extract spatial feature vectors in the face image based on the pre-trained ResNet model, and the spatial feature vectors contain important spatial information in the face image; The feature enhancement module is used to perform feature enhancement on the spatial feature vector extracted by the ResNet model based on the attention mechanism, calculate the attention weight of each time step through the attention mechanism, and apply the attention weight to the original spatial feature vector, so as to obtain the enhanced spatial feature vector; The temporal feature analysis module is used to input the enhanced spatial feature vector extracted from each frame of the face image into the LSTM network in chronological order to obtain the temporal series feature; the LSTM network captures the temporal dependency between video frames through its memory unit and gating mechanism, and analyzes the temporal information in the video; The feature fusion and classification module is used to fuse the time series features output by the LSTM network with the spatial features extracted by the ResNet model to obtain a fused feature vector, input the fused feature vector into the fully connected layer for classification, and input the classification result into the SoftMax layer to calculate the probability of each category to obtain the classification probability of fake and real videos; the fused feature vector contains the temporal and spatial information in the video and can fully reflect the changes in the face in the video.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the multi-dimensional spatiotemporal feature fusion deep fake video detection method based on ResNet and LSTM as described in any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by the processor, the multi-dimensional spatiotemporal feature fusion deep fake video detection method based on ResNet and LSTM as described in any one of claims 1 to 7 is implemented.

Images