Counterfeit video detection method and system based on potential space learning

Abstract

The invention discloses a fake video detection method and system based on potential space learning, and belongs to the technical field of multimedia forensics, and the method comprises the steps: reinforcing the capturing capability of a model for the potential distribution difference of true and false videos through triple contrast learning, combining GRU network modeling time sequence dynamic features and 3D CNN extraction time-space content features, and obtaining a fake video detection result. The joint representation of video time-space domain features is realized, potential spatial features and content features are dynamically fused through an attention mechanism to focus and forge contradictory information, and a global self-attention mechanism of a time encoder is utilized to mine a long-range time sequence dependency relationship. The method overcomes the problems of insufficient generalization ability, weak time sequence modeling and single feature expression of an existing deep counterfeiting detection method, can effectively distinguish a high-quality counterfeiting video from a real video, improves the adaptability to an unknown counterfeiting technology, enhances the detection robustness of a time sequence anomaly and a complex counterfeiting scene, and improves the detection efficiency. And a solution is provided for deep counterfeiting identification.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimedia forensics, and particularly relates to a forged video detection method and system based on latent space learning. Background Art

[0002] With the wide application of deep learning in the fields of computer vision and image processing, Deepfake Detection has become a research hotspot in recent years. Deepfake technology uses advanced models such as Generative Adversarial Networks (GANs) to generate forged content that is almost indistinguishable from real images. This technology has brought many positive applications in movies, entertainment, and social media, but due to its potential malicious uses, especially in aspects such as privacy infringement, it has become an urgent problem to be solved.

[0003] Although existing deepfake detection methods have achieved certain results to a certain extent, they still face many challenges, especially the problem of generalization ability. Existing forgery detection technologies mainly focus on distinguishing real and forged images through detail analysis (such as texture, facial movements, lighting inconsistencies, etc.). However, with the continuous progress of forgery technology, the quality of forged images and videos is getting higher and higher, and it is increasingly difficult to detect details such as forged facial expressions, image textures, and lighting. Therefore, detection algorithms are gradually facing the problem of overfitting, resulting in insufficient generalization ability when facing new types of forged content. Summary of the Invention

[0004] To solve the above technical problems, the present invention proposes a forged video detection method and system based on latent space learning to solve the problems existing in the above prior art.

[0005] In the first aspect, to achieve the above object, the present invention provides a forged video detection method based on latent space learning, including the following steps:

[0006] Process the video data to obtain the face image in a single-frame image, and construct triplet data;

[0007] Extract the feature vectors of the triplet data, obtain the feature stream through the feature vectors, and obtain the latent space embedding vector based on the feature stream;

[0008] Process the triplet data based on the time dimension and the space dimension to obtain the spatio-temporal content features;

[0009] Fuse the latent space embedding vector and the spatio-temporal content features based on the attention mechanism, and input the fused features into the time encoder to obtain the forgery probability.

[0010] Optionally, the process of processing video data to obtain a face image in a single-frame image and constructing triplet data includes:

[0011] The video data includes real videos and forged videos. The real videos and the forged videos are respectively sliced to obtain a continuous number of frames of images. Then, face detection is performed on a single-frame image to generate a face candidate box, and the face area in the face candidate box is cropped to obtain a face image. After the processing is completed, a real video sequence and a forged video sequence are obtained;

[0012] Select several video segments from the real video sequence as reference samples;

[0013] Select several randomly selected video segments from the real video sequence other than the reference samples as positive samples;

[0014] Select several video segments from the forged video sequence as negative samples;

[0015] Construct triplet data based on the reference samples, the positive samples, and the negative samples;

[0016] Among them, the number of video segments selected for the reference samples, the positive samples, and the negative samples is the same, and each video segment contains continuous frame images with the same number of frames.

[0017] Optionally, the process of extracting the feature vector of the triplet data, obtaining the feature stream through the feature vector, and obtaining the latent space embedding vector based on the feature stream includes:

[0018] Extract the feature vector of a single-frame image in the triplet data through an encoder, and perform differential calculation on the feature vectors of adjacent frames to obtain a feature stream;

[0019] Input the feature stream into a GRU network to obtain the temporal change features and output the latent space embedding vector.

[0020] Optionally, the process of processing the triplet data based on the time dimension and the space dimension to obtain the spatio-temporal content features includes:

[0021] Perform convolutional processing on the triplet data based on a 3D convolutional neural network to extract the spatio-temporal content features of the video.

[0022] Optionally, before fusing the latent space embedding vector and the spatio-temporal content features, it further includes:

[0023] Freeze the parameters of the encoder and the GRU network.

[0024] Optionally, based on the attention mechanism, fuse the latent space embedding vector and the spatio-temporal content features, and input the fused features into a time encoder to obtain the forgery probability. The process includes:

[0025] Calculate the correlation between the latent space embedding vector and the spatio-temporal content features based on the attention mechanism to obtain the attention weights;

[0026] Fuse the latent space features and the spatio-temporal content features based on the attention weights;

[0027] Input the fused features into the time encoder to obtain the forgery probability.

[0028] In a second aspect, the present invention also provides a forged video detection system based on latent space learning for implementing a forged video detection method based on latent space learning. The system includes:

[0029] A video processing module for processing video data to obtain face images in single-frame images and constructing triplet data;

[0030] A feature extraction module for extracting feature vectors of the triplet data, obtaining a feature stream through the feature vectors, and obtaining a latent space embedding vector based on the feature stream;

[0031] A spatio-temporal feature processing module for processing the triplet data based on the time dimension and the space dimension to obtain spatio-temporal content features;

[0032] A feature fusion module for fusing the latent space embedding vector and the spatio-temporal content features based on the attention mechanism;

[0033] A forgery detection module for inputting the fused features into the time encoder to obtain the forgery probability.

[0034] Optionally, the video processing module includes:

[0035] A video segmentation unit for respectively slicing the real video and the forged video to obtain a continuous number of frames of images;

[0036] A face detection unit for detecting faces in single-frame images, generating face candidate boxes, cropping to obtain face images, and further obtaining a real video sequence and a forged video sequence;

[0037] A data construction unit for selecting several video segments from the real video sequence as reference samples, randomly selecting several video segments from the real video sequence as positive samples, selecting several video segments from the forged video sequence as negative samples, and constructing triplet data based on the reference samples, positive samples, and negative samples.

[0038] In a third aspect, the present invention also provides a computer terminal device, including:

[0039] One or more processors;

[0040] A memory, coupled to the processor, for storing one or more programs;

[0041] When the one or more programs are executed by the one or more processors, the one or more processors implement a forged video detection method based on latent space learning.

[0042] In a fourth aspect, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, a forged video detection method based on latent space learning is implemented.

[0043] Compared with the prior art, the present invention has the following advantages and technical effects:

[0044] A forged video detection method and system based on latent space learning provided by the present invention strengthen the model's ability to capture the differences in the latent distributions of genuine and forged videos through triplet contrastive learning, combine a GRU network to model temporal dynamic features and 3D CNN to extract spatio-temporal content features to achieve joint representation of video spatio-temporal domain features, and then dynamically fuse latent space features and content features through an attention mechanism to focus on forged contradictory information, and utilize the global self-attention mechanism of the time encoder to mine long-range temporal dependencies. Based on this, the present invention overcomes the problems of insufficient generalization ability, weak temporal modeling, and single feature expression in existing deep forgery detection methods, can effectively distinguish high-quality forged videos from real videos, improve the adaptability to unknown forgery techniques, enhance the detection robustness of temporal anomalies (such as inconsistent expressions) and complex forgery scenarios, and provides a high-precision solution for deep forgery identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings constituting a part of this application are used to provide a further understanding of this application. The schematic embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation to this application. In the drawings:

[0046] Figure 1 is a flowchart of an embodiment of the present invention;

[0047] Figure 2 is a structural diagram of a representation learning module of an embodiment of the present invention;

[0048] Figure 3 is a structural diagram of a forgery detection module of an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] It should be noted that, without conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The following will refer to the drawings and combine the embodiments to detail this application.

[0050] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0051] In view of the deficiencies of the existing methods, the present invention proposes a forged video detection method and system based on latent space learning. This method extracts latent space vectors through contrastive learning to capture subtle features in the images, which are crucial in forgery recognition, and uses GRU to integrate temporal feature information, enabling the model to learn the different temporal and spatial features of real videos and forged videos. Finally, through weighted summation of the spatio-temporal content features and latent space features using the attention mechanism, accurate identification of video forgery is completed.

[0052] Embodiment 1

[0053] As Figure 1 shown, this embodiment provides a forged video detection method based on latent space learning, including:

[0054] Process the video data to obtain the face images in single frames and construct triplet data;

[0055] Extract the feature vectors of the triplet data, obtain a feature stream through the feature vectors, and obtain a latent space embedding vector based on the feature stream;

[0056] Process the triplet data based on the time dimension and the space dimension to obtain spatio-temporal content features;

[0057] Fuse the latent space embedding vector and the spatio-temporal content features based on the attention mechanism, and input the fused features into a time encoder to obtain a forgery probability.

[0058] Specifically, by combining the differences between the latent space feature vectors and content feature vectors of real videos and forged videos, forgery identification is performed, and a good forgery detection effect is achieved.

[0059] S1. Video preprocessing module:

[0060] In the video data preprocessing stage, first use the OpenCV library to extract frames from the video. Each video is segmented into multiple segments for training. Then, face detection is performed through the RetinaFace model, and the face regions are extracted from each frame image. Finally, a triplet sample set (Anchor Clip, Positive Clip, and Negative Clip) is constructed for contrastive learning to improve the discrimination ability of the model in forged video detection.

[0061] S2. Feature Learning Module:

[0062] As Figure 2 shown, in the latent space feature learning stage, first use the Encoder to extract image features for each frame. To capture the changes between video frames, calculate the differences between adjacent frame features to obtain the feature stream , as follows:

[0063] ;

[0064] to provide dynamic information for subsequent learning. Then, use GRU to learn the video latent vector with temporal information , as follows:

[0065] ;

[0066] Among them, is the style feature extraction encoder, is the video clip, represents the latent style features extracted by the encoder. The whole process is pre-trained through contrastive learning, and finally, these latent space vectors can effectively distinguish the inconsistencies in texture features in the spatial and temporal domains between real videos and forged videos.

[0067] In the content feature learning stage, process the input video clip through a 3D convolutional neural network (3D CNN) to extract the content features of the video.

[0068] ;

[0069] where I is the original video clip, represents the 3D convolutional neural network, represents the extracted content features, that is, spatio-temporal content features.

[0070] S3. Forgery Detection Module:

[0071] As Figure 3 shown, this module introduces an attention mechanism to integrate the latent features and spatio-temporal domain features learned by the latent space learning module, and uses a time encoder to integrate the learned information in the time dimension. Finally, output the results of video deep forgery detection through the classification layer.

[0072] The process can be summarized as:

[0073] ;

[0074] Among them, is the classification result, represents the Sigmoid function, is a time encoder, is a style attention module, represents the latent space embedding vector.

[0075] That is, first obtain the training video data, secondly rely on the learned latent space information that is different between real videos and forged videos, fuse the spatio-temporal domain information learned by 3D CNN, and then input it into the neural network through the classification attention network to train the network, and use the trained network to perform forgery discrimination on the video. Compared with the existing forgery video detection, the detection generalization ability is poor. The present invention uses a method of latent space learning to integrate the learned latent features with the spatio-temporal domain information, improving the generalization ability of deepfake video discrimination.

[0076] As an implementation method in this embodiment, the process of processing video data to obtain the face image in a single-frame image and constructing triplet data includes:

[0077] The video data includes real videos and forged videos. The real videos and the forged videos are respectively sliced to obtain a continuous number of frames of images. Then, face detection is performed on a single-frame image to generate face candidate boxes, and the face regions in the face candidate boxes are cropped to obtain face images. After the processing is completed, a real video sequence and a forged video sequence are obtained;

[0078] Select several video clips from the real video sequence as the reference samples;

[0079] Randomly select several video clips from the real video sequence other than the reference samples as the positive samples;

[0080] Select several video clips from the forged video sequence as the negative samples;

[0081] Construct triplet data based on the reference samples, the positive samples, and the negative samples;

[0082] Among them, the number of video clips selected for the reference samples, the positive samples, and the negative samples is the same, and each video clip contains continuous frame images with the same number of frames.

[0083] Specifically, as Figure 1 shown, the processing process in the video data preprocessing module includes:

[0084] 1.1 Frame extraction:

[0085] Use the opencv library to first slice a single video, load each input video, and parse its frame rate and total number of frames. Set the total number of frames of each video to , 8 consecutive frames are selected for training from each slice. To uniformly select a continuous segment in the video, a starting frame position can be randomly or sequentially selected according to the total number of frames in the video. A simple selection method is:

[0086] ;

[0087] where, represents the starting position. In this way, 32 frames are extracted from the middle of the video to ensure that the extracted segment can contain the main content of the video as much as possible. For each clip, 8 consecutive frames are extracted starting from the starting position start_frame, and 4 clip segments are extracted from each video.

[0088] 1.2. Face extraction:

[0089] RetinaFace can detect all face regions in the image, including faces under different angles, illuminations, and poses. The preprocessed clip is input into the RetinaFace model, and face detection is performed on each frame. The model extracts features through a multi-layer convolutional neural network and generates face candidate boxes. According to the bounding boxes detected by RetinaFace, the face regions in the video frames are cropped to remove background interference and focus on the subtle features of the face.

[0090] 1.3. Data organization:

[0091] The present invention proposes a data organization scheme for constructing a triple sample set (i.e., Anchor Clip, Positive Clip, and Negative Clip) to improve the discrimination ability of the model in the Deepfake detection task through contrastive learning.

[0092] Anchor Clip (anchor segment): A video segment selected from a real video sequence is called an AnchorClip. This segment is used to provide the benchmark style features of the real video to ensure a consistent "style" basis in subsequent comparisons. Each Anchor Clip consists of several consecutive video frames, which are extracted with a fixed number of frames and sampling interval to ensure temporal consistency.

[0093] Positive Clip (positive sample segment): Another segment selected from a real video sequence with the same "style" as the Anchor Clip is called a Positive Clip. This segment is used to guide the model to learn the consistency between videos with similar "styles". The Positive Clip has the same "style" features as the Anchor Clip (i.e., both are real video segments), but may come from different positions of the same video or different video sequences.

[0094] Negative Clip (Negative Sample Segment): A video segment selected from a forged video sequence is called a Negative Clip. This segment represents a "style" different from the Anchor Clip and the Negative Clip, and is used to learn the differences between genuine and forged videos. The features of the Negative Clip are significantly different from those of real videos, and are used to guide the model to capture forged feature patterns in contrastive learning.

[0095] Combine the Anchor Clip, Positive Clip, and Negative Clip into a data sample in the form of a triple. Each triple consists of an Anchor Clip, a Positive Clip, and a Negative Clip for contrastive learning during the training process.

[0096] As an optional implementation in this embodiment, there is a strict sequential connection between "frame extraction and face localization" and "triple data construction". The specific process is as follows:

[0097] Frame extraction: Split the original video into continuous frame segments (e.g., 8 frames / Clip) to form basic data units.

[0098] Face localization: Detect the face region in each frame through the RetinaFace model, crop and normalize it (e.g., 256×256 resolution) to eliminate background interference.

[0099] Triple construction: Based on the preprocessed Clip (frames have been extracted and faces have been cropped), select the Anchor Clip, Positive Clip, and Negative Clip from the real video and forged video libraries respectively to form triple samples.

[0100] Data preparation dependencies:

[0101] Each Clip in the triple (whether it is Anchor, Positive, or Negative) must undergo frame extraction and face localization processing to ensure the uniformity of the input data format (e.g., resolution, face region alignment).

[0102] As an optional implementation in this embodiment, the triple includes:

[0103] Anchor Clip (Real Video Segment): As a benchmark, it provides the potential space feature distribution of real videos.

[0104] Positive Clip (Authentic Video Clip): It has the same label (authentic) as the Anchor Clip. By reducing the distance between their latent features, it enhances the model's learning of the feature consistency of authentic videos.

[0105] Negative Clip (Fake Video Clip): It has a different label (fake) from the Anchor Clip. By increasing the distance between their latent features, it forces the model to capture the essential differences between authentic and fake videos.

[0106] As an implementation manner in this embodiment, the process of extracting the feature vectors of the triplet data, obtaining the feature stream through the feature vectors, and obtaining the latent space embedding vector based on the feature stream includes:

[0107] Extract the feature vectors of single-frame images in the triplet data through the encoder, and perform differential calculation on the feature vectors of adjacent frames to obtain the feature stream;

[0108] Input the feature stream into the GRU network to obtain the temporal change features and output the latent space embedding vector.

[0109] Specifically, as Figure 2 shown, the processing process in the representation learning module includes:

[0110] 2.1. Latent Space Representation Learning:

[0111] In the present invention, the Encoder can encode the input image so that its feature vector can represent the multi-level features of the image.

[0112] Input the preprocessed video clip into the encoder, and extract the feature vector of the video clip through this encoder. Through this process, the feature vector is extracted ;

[0113] ;

[0114] where is the d-dimensional style vector of the k-th frame. , is the number of frames extracted.

[0115] The feature vector extracted by the Encoder is only a shallow feature of the picture, so it is not sufficient to be directly applied to the deepfake detection task. To capture the changes between video frames, by performing differential operations on the latent vectors of adjacent frames, the generated "feature stream" can reflect the dynamic change trend in the video sequence. Specifically, the differential vector represents the change amount from the i-th frame to the i+1-th frame. In this way, the entire feature stream It can describe the continuous changes of inter-frame features in video clips, providing more representative features for subsequent model learning.

[0116] To capture the relationships between time series, the sequence is used as input and processed through a GRU module with shared weights to learn the time series change features. The output of the GRU module contains the time series change information between video frames, which can reveal the dynamic features of video clips in the time dimension, thus providing support for further deepfake detection. In the present invention, the hidden state of the last layer of GRU is used to obtain the time embedding style , where represents the size of the hidden unit GRU, B represents the direction of GRU, L represents the number of layers of GRU, and the formula is expressed as follows:

[0117] ;

[0118] represents the gated recurrent unit, and the sequence is used as input and processed through a GRU module with shared weights; in the process of latent space representation learning, different sampling strategies are used to construct reference samples, positive samples, and negative samples. First, the reference stream ( ) is extracted from the segments sampled from the video sequence by the sliding window sampling strategy. On the other hand, the positive sample stream ( ) is obtained by using the random sampling strategy. The reference sample and the positive sample stream are both extracted from the preprocessed segments with the same label. In contrast, the negative sample stream ( ) is obtained from data with labels different from those of the reference and positive samples.

[0119] The reference samples, positive samples, and negative sample streams are sent to the GRU layer, and their representations are trained by using the triplet loss and the classification loss. The triplet loss function for training the GRU layer when defining the final hidden state of GRU from , and is as follows: , and :

[0120] ;

[0121] where, represents the triplet loss function, represents the feature of the true sample output in GRU, represents the feature of the homologous true sample output by GRU, represents the feature of the forged sample output by GRU, Denote the critical hyperparameter. By pre-training the model, the model learns different latent space representations of real videos and forged videos to help the subsequent model better learn forgery discrimination.

[0122] As an implementation manner in this embodiment, the process of processing the triple data based on the time dimension and the space dimension to obtain spatio-temporal content features includes:

[0123] Perform convolution processing on the triple data based on a 3D convolutional neural network to extract the spatio-temporal content features of the video.

[0124] Specifically, the process of obtaining spatio-temporal content features further specifically includes:

[0125] 2.2. Content representation learning:

[0126] As Figure 3 shown, process the input video clip through a 3D convolutional neural network (3D CNN) to extract the content features of the video. The 3D CNN module can perform convolution operations simultaneously in the time and space dimensions to capture the spatio-temporal information in the video. In this way, the video content features extracted by the 3D CNN can effectively represent the actions, objects, and scenes in the video clip, providing rich content representations for deep fake detection. The formula is as follows:

[0127] ;

[0128] where contains inconsistent information in the time sequence and vision in the video clip.

[0129] As an implementation manner in this embodiment, before fusing the latent space embedding vector and the spatio-temporal content features, it further includes:

[0130] Freeze the parameters of the encoder and the GRU network.

[0131] Specifically, the processing process in the forgery detection module includes:

[0132] This module is designed to perform a true / false binary classification task. The present invention uses the traditional binary cross-entropy (BCE) loss to guide the processes of content representation learning, latent space feature fusion, and feature integration in the time dimension, while freezing the weights of each module trained in the first stage.

[0133] As an implementation manner in this embodiment, the process of fusing the latent space embedding vector and the spatio-temporal content features based on the attention mechanism and inputting the fused features into the time encoder to obtain the forgery probability includes:

[0134] Calculate the correlation between the latent space embedding vector and the spatio-temporal content features based on the attention mechanism to obtain the attention weights;

[0135] Fuse the latent space features and the spatio-temporal content features based on the attention weights;

[0136] Input the fused features into the time encoder to obtain the forgery probability.

[0137] Specifically, 3.1. The process of feature fusion includes:

[0138] This part is the Attention Fusion Module (AFM), which introduces an attention mechanism. This module uses the latent space embedding vector obtained from the GRU, and takes as the query, and takes as the key and value, which are expressed as follows:

[0139] ;

[0140] ;

[0141] where represents the softmax function, is the abbreviation of Feature attention, represents the query matrix, 、 、 represent three linear mapping functions, represents the key matrix, represents the value matrix, is the dimension of the key vector, which is used to scale the gradient stability.

[0142] Finally, calculate the result of AFM by taking the dot product of the output of FA and the value component. Specifically, the output of the attention module is calculated as follows:

[0143] ;

[0144] where is the last linear projection layer of AFM, represents the attention calculation function.

[0145] 3.2. Feature integration in the time dimension;

[0146] In the present invention, the output from the feature fusion is linearly projected onto 16 time steps (n = 16), and then input into the Temporal Transformer Encoder (TTE) module. The core of the TTE consists of standard Transformer encoding blocks, each block containing a multi-head self-attention module and a multi-layer perceptron. The output of the TTE serves as the input to the subsequent classification head, and the final deepfake prediction is generated through the sigmoid function The expression is as follows:

[0147] ;

[0148] where is a fully connected layer.

[0149] As an alternative implementation in this embodiment, the core of the attention mechanism is to dynamically focus on key information, and feature fusion is achieved through three steps:

[0150] Feature interaction: Using spatio-temporal content features (such as video action information) as the "questioner" to actively retrieve relevant parts in the latent space features (such as temporal dynamic patterns).

[0151] Weight assignment: Calculate the correlation between the two, assign high weights to key features (such as sudden changes in facial actions), and suppress irrelevant information (such as background noise).

[0152] Feature fusion: Weightedly integrate the key dynamic information in the latent space features (such as inconsistent expressions), and combine it with the spatio-temporal content features (such as abnormal action amplitudes) to form more comprehensive forgery evidence.

[0153] As an alternative implementation in this embodiment, the temporal encoder captures long-range anomalies through global temporal modeling, which is divided into three stages:

[0154] Serialized input: Arrange the fused features in chronological order (such as segmented into 16 time segments) to form a time series.

[0155] Self-attention mechanism: Analyze the relationship between each time segment and all segments (such as an anomaly in the 5th frame being associated with the starting action in the 1st frame).

[0156] Multi-head parallel processing: Capture dependencies from different perspectives through multiple groups of attention heads (such as action continuity, light and shadow changes).

[0157] Anomaly detection: Aggregate the global temporal features, identify long-range anomalies (such as periodic expression repetitions), and output the forgery probability.

[0158] Based on this, a forged video detection method based on latent space learning provided by an embodiment of the present invention strengthens the model's ability to capture the differences in the latent distributions of genuine and forged videos through triplet contrast learning. It combines a GRU network to model temporal dynamic features and a 3D CNN to extract spatio-temporal content features to achieve the joint representation of video spatio-temporal domain features. Then, an attention mechanism is used to dynamically fuse latent space features and content features to focus on forged contradictory information, and the global self-attention mechanism of the time encoder is used to mine long-range temporal dependence relationships. Based on this, the present invention overcomes the problems of insufficient generalization ability, weak temporal modeling, and single feature expression in existing deep forgery detection methods, can effectively distinguish high-quality forged videos from real videos, improve the adaptability to unknown forgery techniques, enhance the detection robustness of temporal anomalies (such as inconsistent expressions) and complex forgery scenarios, and provides a high-precision solution for deep forgery identification.

[0159] Embodiment 2

[0160] In this embodiment, a computer terminal device is provided, including:

[0161] One or more processors;

[0162] A memory, coupled to the processor, for storing one or more programs;

[0163] When the one or more programs are executed by the one or more processors, the one or more processors implement the method in the above embodiments.

[0164] In this embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the method in the above embodiments is implemented.

[0165] In this embodiment, an electronic device is further provided, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the method in the above embodiments.

[0166] The above program can run in a processor or can also be stored in a memory (or referred to as a computer-readable medium). The computer-readable medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0167] These computer programs can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate computer-implemented processing. Thus, the instructions executed on the computer or other programmable device provide for implementing the steps for the functions specified in one process Figure 1 one process or multiple processes and / or boxes Figure 1 steps for the functions specified in one box or multiple boxes. Corresponding to different steps, different modules can be used to implement them.

[0168] In this embodiment, such a device or system is provided. This system is called a forged video detection system based on latent space learning and includes:

[0169] A video processing module for processing video data to obtain a face image in a single-frame image and constructing triplet data;

[0170] A feature extraction module for extracting a feature vector of the triplet data, obtaining a feature stream through the feature vector, and obtaining a latent space embedding vector based on the feature stream;

[0171] A spatio-temporal feature processing module for processing the triplet data based on the time dimension and the space dimension to obtain spatio-temporal content features;

[0172] A feature fusion module for fusing the latent space embedding vector and the spatio-temporal content features based on an attention mechanism;

[0173] A forgery detection module for inputting the fused features into a time encoder to obtain a forgery probability.

[0174] As an implementation manner in this embodiment, the video processing module includes:

[0175] A video segmentation unit for slicing a real video and a forged video respectively to obtain a number of consecutive frames of images;

[0176] A face detection unit for detecting faces in a single frame of image, generating face candidate boxes, cropping to obtain face images, and further obtaining a real video sequence and a forged video sequence;

[0177] A data construction unit for selecting several video segments from the real video sequence as benchmark samples, randomly selecting several video segments from the real video sequence as positive samples, selecting several video segments from the forged video sequence as negative samples, and constructing triplet data based on the benchmark samples, positive samples and negative samples.

[0178] As an implementation manner in this embodiment, the feature extraction module includes:

[0179] An encoder for extracting feature vectors of single frames of image in the triplet data;

[0180] A difference calculation unit for performing difference calculation on the feature vectors of adjacent frames to obtain a feature stream;

[0181] A GRU network unit for processing the feature stream, extracting temporal and spatial variation features, and outputting a latent space embedding vector.

[0182] The spatio-temporal feature processing module includes:

[0183] A 3D convolution processing unit for performing convolution processing on the triplet data based on a 3D convolutional neural network to extract spatio-temporal content features of the video.

[0184] As an implementation manner in this embodiment, before fusing the latent space embedding vector and the spatio-temporal content features, the feature fusion module further includes:

[0185] A parameter freezing unit for freezing the parameters of the encoder and the GRU network.

[0186] As an implementation manner in this embodiment, the feature fusion module includes:

[0187] An attention calculation unit for calculating the correlation between the latent space embedding vector and the spatio-temporal content features based on an attention mechanism to obtain attention weights;

[0188] A feature fusion unit for fusing the latent space features and the spatio-temporal content features based on the attention weights.

[0189] The system or device is used to implement the functions of the methods in the above embodiments. Each module in the system or device corresponds to each step in the method, and those that have been described in the method will not be elaborated here.

[0190] Through the above implementation manner, the problem of fake video detection based on latent space learning in the related art is solved. Based on this, the present invention overcomes the problems of insufficient generalization ability, weak temporal modeling, and single feature expression in the existing deep fake detection methods, can effectively distinguish high-quality fake videos from real videos, improve the adaptability to unknown forgery techniques, enhance the detection robustness of temporal anomalies (such as inconsistent expressions) and complex forgery scenarios, and provides a high-precision solution for deep fake identification.

[0191] The above is only a preferred specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for detecting forged videos based on latent space learning, characterized in that, It includes the following steps: Process the video data to obtain a face image in a single-frame image, and construct triplet data; Extract the feature vectors of the triplet data, obtain a feature stream through the feature vectors, and obtain a latent space embedding vector based on the feature stream; Process the triplet data based on the time dimension and the space dimension to obtain spatio-temporal content features; Fuse the latent space embedding vector and the spatio-temporal content features based on the attention mechanism, and input the fused features into a time encoder to obtain a forgery probability.

2. The method according to claim 1, wherein The process of processing the video data to obtain a face image in a single-frame image and constructing triplet data includes: The video data includes a real video and a forged video. The real video and the forged video are respectively sliced to obtain a continuous number of frames of images. Then, face detection is performed on a single-frame image to generate a face candidate box, and the face area in the face candidate box is cropped to obtain a face image. After the processing is completed, a real video sequence and a forged video sequence are obtained; Select several video segments from the real video sequence as reference samples; Select several randomly selected video segments from the real video sequence other than the reference samples as positive samples; Select several video segments from the forged video sequence as negative samples; Construct triplet data based on the reference samples, the positive samples, and the negative samples; Among them, the number of video segments selected for the reference samples, the positive samples, and the negative samples is the same, and each video segment contains continuous frame images with the same number of frames.

3. The method according to claim 2, characterized in that, The process of extracting the feature vectors of the triplet data, obtaining a feature stream through the feature vectors, and obtaining a latent space embedding vector based on the feature stream includes: Extract the feature vectors of single-frame images in the triplet data through an encoder, and perform differential calculation on the feature vectors of adjacent frames to obtain a feature stream; Input the feature stream into a GRU network to obtain temporal variation features and output a latent space embedding vector.

4. The method according to claim 1, characterized in that The process of processing the triplet data based on the time dimension and the space dimension to obtain spatio-temporal content features includes: Perform convolutional processing on the triplet data based on a 3D convolutional neural network to extract spatio-temporal content features of the video.

5. The method according to claim 3, wherein Before fusing the latent space embedding vector and the spatio-temporal content features, it also includes: Freeze the parameters of the encoder and the GRU network.

6. The method according to claim 1, wherein The process of fusing the latent space embedding vector and the spatio-temporal content features based on the attention mechanism, and inputting the fused features into a time encoder to obtain a forgery probability includes: Calculate the correlation between the latent space embedding vector and the spatio-temporal content features based on the attention mechanism to obtain attention weights; Fuse the latent space features and the spatio-temporal content features based on the attention weights; Input the fused features into a time encoder to obtain a forgery probability.

7. A forged video detection system based on latent space learning, characterized in that, The system includes: A video processing module for processing video data to obtain a face image in a single-frame image and constructing triplet data; A feature extraction module for extracting the feature vectors of the triplet data, obtaining a feature stream through the feature vectors, and obtaining a latent space embedding vector based on the feature stream; A spatio-temporal feature processing module, configured to process triple data based on the time dimension and the space dimension to obtain spatio-temporal content features; A feature fusion module, configured to fuse the latent space embedding vector and the spatio-temporal content features based on an attention mechanism; A forgery detection module, configured to input the fused features into a time encoder to obtain a forgery probability.

8. The system according to claim 7, characterized in that, The video processing module includes: A video segmentation unit, configured to slice a real video and a forged video respectively to obtain a plurality of consecutive frames of images; A face detection unit, configured to perform face detection on a single frame of image, generate a face candidate box, crop to obtain a face image, and further obtain a real video sequence and a forged video sequence; A data construction unit, configured to select several video segments from the real video sequence as reference samples, randomly select several video segments from the real video sequence as positive samples, select several video segments from the forged video sequence as negative samples, and construct triple data based on the reference samples, positive samples and negative samples.

9. A computer terminal device, characterized in that, including: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the forged video detection method based on latent space learning according to any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the forged video detection method based on latent space learning according to any one of claims 1-6.

Images