Forged video detection method and system based on latent space learning

Abstract

The present invention discloses a method and system for detecting forged videos based on latent space learning, which belongs to the field of multimedia forensics technology. The present invention strengthens the model's ability to capture the potential distribution differences between authentic and fake videos through triple contrastive learning, combines the temporal dynamic features of the GRU network modeling with the spatiotemporal content features extracted by 3D CNN, and realizes the joint representation of the spatiotemporal domain features of the video. The latent space features and content features are then dynamically integrated through the attention mechanism to focus on forged contradictory information, and the global self-attention mechanism of the temporal encoder is used to mine long-range temporal dependencies. The present invention overcomes the problems of insufficient generalization ability, weak temporal modeling, and single feature expression of existing deep forgery detection methods. It can effectively distinguish high-quality forged videos from real videos, improve adaptability to unknown forgery techniques, and enhance the detection robustness of temporal anomalies and complex forgery scenes, providing a solution for deep forgery identification.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimedia forensics, and in particular relates to a forged video detection method and system based on latent space learning. Background Art

[0002] With the widespread application of deep learning in computer vision and image processing, deepfake detection has become a research hotspot in recent years. Deepfake technology utilizes advanced models such as generative adversarial networks (GANs) to generate fake content that is virtually indistinguishable from real images. While this technology has many positive applications in film, entertainment, and social media, it also poses urgent challenges due to its potential for malicious use, particularly in privacy violations.

[0003] While existing deepfake detection methods have achieved some success, they still face numerous challenges, particularly generalization. Existing forgery detection techniques primarily focus on distinguishing real from forged images through detailed analysis, such as texture, facial movements, and lighting inconsistencies. However, with the continuous advancement of forgery technology, the quality of forged images and videos has increased, making details like forged facial expressions, image texture, and lighting increasingly difficult to detect. Consequently, detection algorithms are increasingly overfitting, resulting in insufficient generalization when faced with new types of forged content. Summary of the Invention

[0004] To solve the above technical problems, the present invention proposes a method and system for detecting forged videos based on latent space learning to solve the problems existing in the above-mentioned prior art.

[0005] In a first aspect, to achieve the above-mentioned objectives, the present invention provides a method for detecting forged videos based on latent space learning, comprising the following steps:

[0006] Process the video data to obtain the face image in a single frame image and construct triplet data;

[0007] Extracting a feature vector of the triple data, obtaining a feature stream through the feature vector, and obtaining a latent space embedding vector based on the feature stream;

[0008] Process the triplet data based on the time dimension and space dimension to obtain spatiotemporal content features;

[0009] The latent space embedding vector and the spatiotemporal content feature are fused based on an attention mechanism, and the fused feature is input into a temporal encoder to obtain a forgery probability.

[0010] Optionally, the process of processing the video data to obtain a face image in a single frame image and constructing triplet data includes:

[0011] The video data includes real videos and forged videos, and the real videos and the forged videos are sliced ​​to obtain a plurality of consecutive frames of images. Then, face detection is performed on the single-frame images to generate face candidate frames, and the face area in the face candidate frames is cropped to obtain face images. After the processing is completed, real video sequences and forged video sequences are obtained.

[0012] Several video clips are selected from real video sequences as benchmark samples;

[0013] Several video clips randomly selected from the real video sequence except the benchmark samples are used as positive samples;

[0014] Several video clips selected from the fake video sequence are used as negative samples;

[0015] Constructing triplet data based on the reference sample, the positive sample and the negative sample;

[0016] The reference sample, the positive sample, and the negative sample are selected from the same number of video clips, and each video clip contains the same number of continuous frame images.

[0017] Optionally, extracting a feature vector of the triple data, obtaining a feature stream through the feature vector, and obtaining a latent space embedding vector based on the feature stream includes:

[0018] The encoder extracts the feature vector of a single frame image from the triplet data, and performs differential calculation on the feature vectors of adjacent frames to obtain the feature flow;

[0019] The feature stream is input into the GRU network to obtain the temporal variation features and output the latent space embedding vector.

[0020] Optionally, the process of processing triplet data based on the time dimension and the space dimension to obtain spatiotemporal content features includes:

[0021] The triplet data is convolved based on a 3D convolutional neural network to extract the spatiotemporal content features of the video.

[0022] Optionally, before fusing the latent space embedding vector and the spatiotemporal content feature, the method further includes:

[0023] Freeze the parameters of the encoder and the GRU network.

[0024] Optionally, the process of fusing the latent space embedding vector and the spatiotemporal content feature based on an attention mechanism, inputting the fused feature into a temporal encoder, and obtaining the forgery probability includes:

[0025] Based on the attention mechanism, the correlation between the latent space embedding vector and the spatiotemporal content features is calculated to obtain the attention weight;

[0026] Based on the attention weight, the latent space features and spatiotemporal content features are integrated;

[0027] The fused features are input into a temporal encoder to obtain the forgery probability.

[0028] In a second aspect, the present invention further provides a forged video detection system based on latent space learning, which is used to implement a forged video detection method based on latent space learning. The system includes:

[0029] The video processing module is used to process the video data, obtain the face image in the single frame image, and construct triplet data;

[0030] A feature extraction module is used to extract feature vectors of triple data, obtain feature streams through the feature vectors, and obtain a latent space embedding vector based on the feature streams;

[0031] The spatiotemporal feature processing module is used to process the triplet data based on the time dimension and the space dimension to obtain spatiotemporal content features;

[0032] A feature fusion module, configured to fuse the latent space embedding vector and the spatiotemporal content features based on an attention mechanism;

[0033] The forgery detection module is used to input the fused features into the temporal encoder to obtain the forgery probability.

[0034] Optionally, the video processing module includes:

[0035] The video segmentation unit is used to slice the real video and the fake video respectively to obtain a number of consecutive frame images;

[0036] The face detection unit is used to perform face detection on a single frame image, generate a face candidate frame, and crop the face image to obtain a real video sequence and a forged video sequence;

[0037] A data construction unit is used to select several video clips from the real video sequence as reference samples, randomly select several video clips from the real video sequence as positive samples, select several video clips from the forged video sequence as negative samples, and construct triplet data based on the reference samples, positive samples and negative samples.

[0038] In a third aspect, the present invention further provides a computer terminal device, comprising:

[0039] one or more processors;

[0040] a memory, coupled to the processor, for storing one or more programs;

[0041] When the one or more programs are executed by the one or more processors, the one or more processors implement a forged video detection method based on latent space learning.

[0042] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for detecting forged videos based on latent space learning.

[0043] Compared with the prior art, the present invention has the following advantages and technical effects:

[0044] The present invention provides a method and system for detecting forged videos based on latent space learning. This method uses triplet contrast learning to enhance the model's ability to capture the potential distribution differences between authentic and fake videos. It then combines GRU network modeling of temporal dynamic features with 3D CNN to extract spatiotemporal content features, achieving a joint representation of video spatiotemporal domain features. It then dynamically fuses latent space features with content features through an attention mechanism to focus on forged contradictory information, and utilizes the global self-attention mechanism of the temporal encoder to mine long-range temporal dependencies. Based on this, the present invention overcomes the problems of existing deepfake detection methods, such as insufficient generalization, weak temporal modeling, and single feature expression. It can effectively distinguish high-quality forged videos from real videos, improve adaptability to unknown forgery techniques, and enhance the detection robustness of temporal anomalies (such as incoherent expressions) and complex forgery scenes, providing a high-precision solution for deepfake identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0046] Figure 1 is a flow chart of an embodiment of the present invention;

[0047] Figure 2 This is a structural diagram of a representation learning module according to an embodiment of the present invention;

[0048] Figure 3 FIG. 4 is a structural diagram of a counterfeit detection module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0050] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0051] In light of the shortcomings of existing methods, this paper proposes a method and system for detecting forged videos based on latent space learning. This method extracts latent space vectors through contrastive learning, capturing subtle image features crucial for forgery detection. It then integrates temporal feature information using a GRU, enabling the model to learn the distinct temporal and spatial characteristics of authentic and forged videos. Finally, by applying a weighted summation of the spatiotemporal content features and the latent space features using an attention mechanism, accurate video forgery identification is achieved.

[0052] Example 1

[0053] like Figure 1 As shown, this embodiment provides a method for detecting forged videos based on latent space learning, including:

[0054] Process the video data to obtain the face image in a single frame image and construct triplet data;

[0055] Extracting a feature vector of the triple data, obtaining a feature stream through the feature vector, and obtaining a latent space embedding vector based on the feature stream;

[0056] Process the triplet data based on the time dimension and space dimension to obtain spatiotemporal content features;

[0057] The latent space embedding vector and the spatiotemporal content feature are fused based on an attention mechanism, and the fused feature is input into a temporal encoder to obtain a forgery probability.

[0058] Specifically, by combining the differences in latent space feature vectors and content feature vectors between real videos and forged videos, forgery identification is performed, and a good forgery identification effect is achieved.

[0059] S1, video preprocessing module:

[0060] During video data preprocessing, OpenCV is used to extract video frames. Each video is then divided into multiple clips for training. Face detection is then performed using the RetinaFace model, extracting the face region from each frame. Finally, a set of triplet samples (anchor clip, positive clip, and negative clip) is constructed for comparative learning, improving the model's ability to distinguish forged videos.

[0061] S2, representation learning module:

[0062] like Figure 2 As shown in the latent space representation learning stage, the Encoder encoder is first used to extract image features for each frame. In order to capture the changes between video frames, the differences in features of adjacent frames are calculated to obtain the feature flow ,as follows:

[0063] ;

[0064] Provide dynamic information for subsequent learning. Then use GRU to learn the video potential vector with temporal information. ,as follows:

[0065] ;

[0066] in, Extract encoder for style features, For video clips, Represents the latent style features extracted by the encoder. The entire process is pre-trained through contrastive learning. Finally, these latent space vectors can effectively distinguish the inconsistencies in texture features in the spatial and temporal domains between real and fake videos.

[0067] In the content representation learning stage, the input video clips are processed by a 3D convolutional neural network (3D CNN) to extract the content features of the video.

[0068] ;

[0069] Where I is the original video clip, represents a 3D convolutional neural network, Represents the extracted content features, that is, spatiotemporal content features.

[0070] S3, Forgery Detection Module:

[0071] like Figure 3 As shown in the figure, this module introduces an attention mechanism to integrate the latent features and spatiotemporal features learned by the latent space learning module, and uses a temporal encoder to integrate the learned information in the temporal dimension, and finally outputs the results of video deep fake detection through the classification layer.

[0072] The process can be summarized as:

[0073] ;

[0074] in, is the classification result, represents the Sigmoid function, is the time encoder, is the style attention module, represents the latent space embedding vector.

[0075] First, training video data is acquired. Then, relying on the learned latent space information of real and forged videos, the spatiotemporal information learned by the 3D CNN is integrated. This information is then fed into a neural network through a classification attention network to train the network. The trained network then identifies forged videos. Compared to existing forged video detection methods, which have limited generalization capabilities, this invention utilizes a latent space learning method to integrate learned latent features with spatiotemporal information, improving the generalization capabilities of deepfake video identification.

[0076] As an implementation method of this embodiment, the process of processing video data to obtain a face image in a single frame image and constructing triplet data includes:

[0077] The video data includes real videos and forged videos, and the real videos and the forged videos are sliced ​​to obtain a plurality of consecutive frames of images. Then, face detection is performed on the single-frame images to generate face candidate frames, and the face area in the face candidate frames is cropped to obtain face images. After the processing is completed, real video sequences and forged video sequences are obtained.

[0078] Several video clips are selected from real video sequences as benchmark samples;

[0079] Several video clips randomly selected from the real video sequence except the benchmark samples are used as positive samples;

[0080] Select several video clips from the fake video sequence as negative samples;

[0081] Constructing triplet data based on the reference sample, the positive sample and the negative sample;

[0082] The reference sample, the positive sample, and the negative sample are selected from the same number of video clips, and each video clip contains the same number of continuous frame images.

[0083] Specifically, such as Figure 1 As shown, the processing process in the video data preprocessing module includes:

[0084] 1.1 Frame extraction:

[0085] Use the opencv library to slice a single video, load each input video, and parse its frame rate and total number of frames. Set the total number of frames for each video to , each slice selects 8 consecutive frames for training. In order to evenly select a continuous segment in the video, a starting frame position can be selected randomly or sequentially according to the total number of frames in the video. A simple selection method is:

[0086] ;

[0087] in, Indicates the starting position, so that 32 frames are extracted from the middle of the video to ensure that the extracted segments can contain the main content of the video as much as possible. For each clip, starting from the starting position start_frame, 8 frames are extracted continuously, and 4 clip segments are extracted from each video.

[0088] 1.2 Face Extraction

[0089] RetinaFace can detect all facial regions in an image, including faces from varying angles, lighting, and poses. The preprocessed clip is fed into the RetinaFace model, which performs face detection on each frame. The model extracts features using a multi-layer convolutional neural network and generates candidate face frames. Based on the bounding boxes detected by RetinaFace, the face region in the video frame is cropped to remove background interference and focus on the subtle features of the face.

[0090] 1.3 Data Organization

[0091] This paper proposes a data organization scheme for constructing a triplet sample set (i.e., Anchor Clip, Positive Clip, and Negative Clip), and improves the model's discrimination ability in Deepfake detection tasks through comparative learning.

[0092] Anchor Clip: A video clip selected from a real video sequence is called an AnchorClip. This clip provides baseline style characteristics of the real video, ensuring a consistent "style" foundation for subsequent comparisons. Each Anchor Clip consists of several consecutive video frames, extracted using a fixed frame count and sampling interval to ensure temporal consistency.

[0093] Positive Clip: A clip selected from a real video sequence with the same "style" as the anchor clip is called a positive clip. This clip is used to guide the model to learn consistency between videos with similar "styles." Positive Clips and Anchor Clips share the same "style" characteristics (i.e., they are both real video clips), but may come from different locations in the same video or different video sequences.

[0094] Negative clips: A video clip selected from a forged video sequence is called a negative clip. This clip represents a different "style" from the anchor clip and negative clips and is used to learn the differences between real and fake videos. The features of the negative clip differ significantly from those of the real video, guiding the model to capture the characteristic patterns of forgeries in contrastive learning.

[0095] Anchor Clip, Positive Clip, and Negative Clip are combined into a data sample in the form of triplets. Each triplet consists of an anchor clip, a positive clip, and a negative clip, so that comparative learning can be performed during training.

[0096] As an optional implementation in this embodiment, there is a strict sequential relationship between "frame extraction and face positioning" and "triple data construction". The specific process is as follows:

[0097] Frame extraction: Divide the original video into continuous frame segments (such as 8 frames / Clip) to form basic data units.

[0098] Face localization: The RetinaFace model is used to detect the face region in each frame, crop it, and standardize it (e.g., to 256×256 resolution) to eliminate background interference.

[0099] Triplet construction: Based on the preprocessed clips (frames have been extracted and faces have been cropped), anchor clips, positive clips, and negative clips are selected from the real video and fake video libraries respectively, and combined into triplet samples.

[0100] Data preparation depends on:

[0101] Each Clip in the triplet (whether Anchor, Positive or Negative) must undergo frame extraction and face positioning processing to ensure that the format of the input data is unified (such as resolution and face area alignment).

[0102] As an optional implementation in this embodiment, the triplet includes:

[0103] Anchor Clip (real video clip): As a benchmark, it provides the latent spatial feature distribution of real videos.

[0104] Positive Clip (real video clip from the same source): has the same label (real) as the Anchor Clip. By narrowing the potential feature distance between the two, the model's learning of the consistency of real video features is enhanced.

[0105] Negative Clip (forged video clip): Different from the Anchor Clip label (forged), it forces the model to capture the essential differences between real and fake videos by pushing its potential feature distance.

[0106] As an implementation method of this embodiment, a process of extracting a feature vector of triple data, obtaining a feature stream through the feature vector, and obtaining a latent space embedding vector based on the feature stream includes:

[0107] The encoder extracts the feature vector of a single frame image from the triplet data, and performs differential calculation on the feature vectors of adjacent frames to obtain the feature flow;

[0108] The feature stream is input into the GRU network to obtain the temporal variation features and output the latent space embedding vector.

[0109] Specifically, such as Figure 2 As shown in Figure 2, the processing in the representation learning module includes:

[0110] 2.1. Latent Space Representation Learning:

[0111] The encoder in the present invention can encode the input image so that its feature vector can represent the multi-level features of the image.

[0112] The pre-processed video clips Input to the encoder, through which the feature vector of the video clip is extracted. Through this process, the feature vector is extracted ;

[0113] ;

[0114] in for The d-dimensional style vector of the k-th frame. , is the number of frames extracted.

[0115] The feature vector extracted by the encoder is only a shallow feature of the image, so it is not enough to be directly applied to the deepfake detection task. In order to capture the changes between video frames, the latent vectors of adjacent frames are differentially calculated to generate a "feature flow" that can reflect the dynamic change trend in the video sequence. Specifically, the differential vector It represents the change from the i-th frame to the i+1-th frame. It can describe the continuous changes of inter-frame features in video clips and provide more representative features for subsequent model learning.

[0116] In order to capture the relationship between time series, The sequence is used as input and processed by the GRU module with shared weights to learn the temporal change features. The output of the GRU module contains the temporal change information between video frames, which can reveal the dynamic characteristics of the video clips in the time dimension, thus providing support for further deep fake detection. The present invention obtains the temporal embedding style by converting the hidden state of the last layer of GRU into a temporal embedding style. ,in Represents the size of the hidden unit GRU, B represents the direction of GRU, and L represents the number of layers of GRU. The formula is as follows:

[0117] ;

[0118] represents the gated recurrent unit, The sequence is taken as input and processed by the GRU module with shared weights. In the process of learning the latent space representation, different sampling strategies are used to construct the reference samples, positive samples, and negative samples. First, the reference stream ( On the other hand, the positive sample flow ( ) is obtained by using a random sampling strategy. Both the baseline samples and the positive sample stream are extracted from the preprocessed segments with the same label. In contrast, the negative sample stream ( ) is obtained from data with labels different from the baseline and positive samples.

[0119] The reference samples, positive samples, and negative sample streams are fed into the GRU layer, and their representations are trained using triplet loss and classification loss. 、 and Define GRU from 、 and When the final hidden state is , the triplet loss function for training the GRU layer is as follows:

[0120] ;

[0121] in, represents the ternary loss function, Represents the true sample features output in GRU, Represents the homologous real sample features output by GRU, Represents the forged sample features output by GRU, Represents the critical hyperparameter. By pre-training the model, the model learns different latent space representations for real and forged videos, helping subsequent models better learn forgery identification.

[0122] As an implementation method of this embodiment, the process of processing triple data based on the time dimension and the space dimension to obtain spatiotemporal content features includes:

[0123] The triplet data is convolved based on a 3D convolutional neural network to extract the spatiotemporal content features of the video.

[0124] Specifically, the process of obtaining spatiotemporal content features includes:

[0125] 2.2 Content Representation Learning

[0126] like Figure 3 As shown in Figure 1, a 3D convolutional neural network (3D CNN) processes the input video clip to extract video content features. The 3D CNN module performs convolution operations simultaneously in both time and space, capturing the spatiotemporal information in the video. In this way, the video content features extracted by the 3D CNN can effectively represent the actions, objects, and scenes in the video clip, providing rich content representation for deepfake detection. The formula is as follows:

[0127] ;

[0128] in It contains information about the temporal and visual inconsistencies in the video clips.

[0129] As an implementation manner in this embodiment, before fusing the latent space embedding vector and the spatiotemporal content feature, the method further includes:

[0130] Freeze the parameters of the encoder and the GRU network.

[0131] Specifically, the processing in the forgery detection module includes:

[0132] This module is designed to perform true or false binary classification tasks. The present invention uses the traditional binary cross entropy (BCE) loss to guide the process of content representation learning, latent space feature fusion and temporal dimension feature integration, while freezing the weights of each module trained in the first stage.

[0133] As an implementation method in this embodiment, the process of fusing the latent space embedding vector and the spatiotemporal content features based on the attention mechanism, inputting the fused features into the temporal encoder, and obtaining the forgery probability includes:

[0134] Based on the attention mechanism, the correlation between the latent space embedding vector and the spatiotemporal content features is calculated to obtain the attention weight;

[0135] Based on the attention weight, the latent space features and spatiotemporal content features are integrated;

[0136] The fused features are input into a temporal encoder to obtain the forgery probability.

[0137] Specifically, 3.1, the feature fusion process includes:

[0138] This part is the Attention Fusion Module (AFM), which introduces an attention mechanism. This module uses the latent space embedding vector obtained from GRU to As a query, As keys and values, represented as follows:

[0139] ;

[0140] ;

[0141] in represents the softmax function, It is the abbreviation of Feature attention. represents the query matrix, 、 、 Represents three linear mapping functions, represents the bond matrix, represents the value matrix, is the dimension of the key vector, used for scaling gradient stability.

[0142] Finally, the result of AFM is calculated by taking the dot product of the output of FA and the value component. Specifically, the output of the attention module is calculated as follows:

[0143] ;

[0144] in is the last linear projection layer of AFM, Represents the attention calculation function.

[0145] 3.2, Feature integration in the time dimension;

[0146] In this paper, the output from feature fusion is linearly projected to 16 time steps (n=16) and then input into the temporal encoder (TTE) module. The core of TTE consists of a standard transformer encoding block, each of which contains a multi-head self-attention module and a multi-layer perceptron. The output of TTE is used as the input of the subsequent classification head and is processed through the sigmoid function. Generate the final deepfake prediction. The expression is as follows:

[0147] ;

[0148] in is the fully connected layer.

[0149] As an optional implementation in this embodiment, the core of the attention mechanism is to dynamically focus on key information and achieve feature fusion through three steps:

[0150] Feature interaction: Using spatiotemporal content features (such as video motion information) as “questioners” to actively retrieve relevant parts of latent spatial features (such as temporal dynamic patterns).

[0151] Weight allocation: Calculate the correlation between the two, assign high weights to key features (such as sudden changes in facial movements), and suppress irrelevant information (such as background noise).

[0152] Feature fusion: Weighted integration of key dynamic information (such as incoherent expressions) in latent space features, combined with spatiotemporal content features (such as abnormal movement amplitude) to form more comprehensive forgery evidence.

[0153] As an optional implementation in this embodiment, the time encoder captures long-range anomalies through global time series modeling, which is divided into three stages:

[0154] Serialized input: Arrange the fused features in chronological order (e.g., divide them into 16 time segments) to form a time series.

[0155] Self-attention mechanism: Analyzes the relationship between each time segment and all segments (e.g., the abnormal correlation between the 5th frame and the starting action of the 1st frame).

[0156] Multi-head parallel processing: Capture dependencies from different perspectives (such as action continuity and light and shadow changes) through multiple sets of attention heads.

[0157] Anomaly detection: Aggregates global temporal features, identifies long-range anomalies (such as periodic expression repetitions), and outputs the probability of forgery.

[0158] Based on this, an embodiment of the present invention provides a method for detecting fake videos based on latent space learning. It uses triple contrast learning to enhance the model's ability to capture the potential distribution differences between real and fake videos, combines the GRU network to model temporal dynamic features with the 3D CNN to extract spatiotemporal content features, and achieves a joint representation of video spatiotemporal domain features. It then dynamically fuses latent space features and content features through an attention mechanism to focus on forged contradictory information, and uses the global self-attention mechanism of the temporal encoder to mine long-range temporal dependencies. Based on this, the present invention overcomes the problems of insufficient generalization ability, weak temporal modeling, and single feature expression of existing deep fake detection methods. It can effectively distinguish high-quality fake videos from real videos, improve adaptability to unknown forgery techniques, and enhance the detection robustness of temporal anomalies (such as incoherent expressions) and complex forgery scenes, providing a high-precision solution for deep fake identification.

[0159] Example 2

[0160] In this embodiment, a computer terminal device is provided, including:

[0161] one or more processors;

[0162] a memory, coupled to the processor, for storing one or more programs;

[0163] When the one or more programs are executed by the one or more processors, the one or more processors implement the methods in the above embodiments.

[0164] In this embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the method in the above embodiment is implemented.

[0165] In this embodiment, an electronic device is further provided, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to execute the method in the above embodiment.

[0166] The above program can be executed in a processor or stored in a memory (or computer-readable medium). Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and can be implemented using any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0167] These computer programs can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps of the functions specified in one or more blocks can be implemented by different modules corresponding to different steps.

[0168] This embodiment provides such a device or system. The system is called a forged video detection system based on latent space learning, and includes:

[0169] The video processing module is used to process the video data, obtain the face image in the single frame image, and construct triplet data;

[0170] A feature extraction module is used to extract feature vectors of triple data, obtain feature streams through the feature vectors, and obtain a latent space embedding vector based on the feature streams;

[0171] The spatiotemporal feature processing module is used to process the triplet data based on the time dimension and the space dimension to obtain spatiotemporal content features;

[0172] A feature fusion module, configured to fuse the latent space embedding vector and the spatiotemporal content features based on an attention mechanism;

[0173] The forgery detection module is used to input the fused features into the temporal encoder to obtain the forgery probability.

[0174] As an implementation manner in this embodiment, the video processing module includes:

[0175] The video segmentation unit is used to slice the real video and the forged video respectively to obtain a number of consecutive frame images;

[0176] The face detection unit is used to perform face detection on a single frame image, generate a face candidate frame, and crop the face image to obtain a real video sequence and a forged video sequence;

[0177] A data construction unit is used to select several video clips from the real video sequence as reference samples, randomly select several video clips from the real video sequence as positive samples, select several video clips from the forged video sequence as negative samples, and construct triplet data based on the reference samples, positive samples and negative samples.

[0178] As an implementation method of this embodiment, the feature extraction module includes:

[0179] An encoder, used to extract the feature vector of a single frame image from the triplet data;

[0180] A differential calculation unit, used to perform differential calculation on the feature vectors of adjacent frames to obtain a feature flow;

[0181] The GRU network unit is used to process the feature stream, extract temporal variation features, and output a latent space embedding vector.

[0182] The spatiotemporal feature processing module includes:

[0183] The 3D convolution processing unit is used to perform convolution processing on the triplet data based on the 3D convolutional neural network to extract the spatiotemporal content features of the video.

[0184] As an implementation method of this embodiment, the feature fusion module further includes:

[0185] A parameter freezing unit is used to freeze the parameters of the encoder and the GRU network.

[0186] As an implementation method of this embodiment, the feature fusion module includes:

[0187] Attention calculation unit, which is used to calculate the correlation between the latent space embedding vector and the spatiotemporal content features based on the attention mechanism to obtain the attention weight;

[0188] The feature fusion unit is used to fuse the latent space features and spatiotemporal content features based on the attention weight.

[0189] The system or device is used to implement the functions of the method in the above-mentioned embodiment. Each module in the system or device corresponds to each step in the method, which has been explained in the method and will not be repeated here.

[0190] Through the above-mentioned implementation, the problem of forged video detection based on latent space learning in the related art is solved. Based on this, the present invention overcomes the problems of insufficient generalization ability, weak temporal modeling, and single feature expression of existing deep fake detection methods. It can effectively distinguish high-quality forged videos from real videos, improve adaptability to unknown forgery techniques, and enhance the detection robustness of temporal anomalies (such as incoherent expressions) and complex forgery scenes, providing a high-precision solution for deep fake identification.

[0191] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for detecting forged videos based on latent space learning, characterized in that: The following steps are involved: The video data includes real videos and forged videos, and the real videos and the forged videos are sliced ​​to obtain a plurality of consecutive frames of images. Then, face detection is performed on the single-frame images to generate face candidate frames, and the face area in the face candidate frames is cropped to obtain face images. After the processing is completed, real video sequences and forged video sequences are obtained. Several video clips are selected from real video sequences as benchmark samples; Several video clips randomly selected from the real video sequence except the benchmark samples are used as positive samples; Several video clips selected from the fake video sequence are used as negative samples; Constructing triplet data based on the reference sample, the positive sample and the negative sample; The reference sample, the positive sample, and the negative sample are selected from the same number of video clips, and each video clip contains the same number of continuous frame images; The encoder extracts the feature vector of a single frame image from the triplet data, and performs differential calculation on the feature vectors of adjacent frames to obtain the feature flow; Input the feature stream into the GRU network to obtain temporal variation features and output a latent space embedding vector; Process the triplet data based on the time dimension and space dimension to obtain spatiotemporal content features; Based on the attention mechanism, the correlation between the latent space embedding vector and the spatiotemporal content features is calculated to obtain the attention weight; Based on the attention weight, the latent space features and spatiotemporal content features are integrated; The fused features are input into a temporal encoder to obtain the forgery probability.

2. The method according to claim 1, characterized in that The process of processing triplet data based on the time dimension and space dimension to obtain spatiotemporal content features includes: The triplet data is convolved based on a 3D convolutional neural network to extract the spatiotemporal content features of the video.

3. The method according to claim 1, characterized in that Before fusing the latent space embedding vector and the spatiotemporal content feature, the method further includes: Freeze the parameters of the encoder and the GRU network.

4. A forged video detection system based on latent space learning, characterized in that The system is used to implement the various steps of the method according to claims 1 to 3, and the system includes: The video processing module is used to process the video data, obtain the face image in the single frame image, and construct triplet data; A feature extraction module is used to extract feature vectors of triple data, obtain feature streams through the feature vectors, and obtain a latent space embedding vector based on the feature streams; The spatiotemporal feature processing module is used to process the triplet data based on the time dimension and the space dimension to obtain spatiotemporal content features; A feature fusion module, configured to fuse the latent space embedding vector and the spatiotemporal content features based on an attention mechanism; The forgery detection module is used to input the fused features into the temporal encoder to obtain the forgery probability.

5. The system according to claim 4, characterized in that The video processing module includes: The video segmentation unit is used to slice the real video and the fake video respectively to obtain a number of consecutive frame images; The face detection unit is used to perform face detection on a single frame image, generate a face candidate frame, and crop the face image to obtain a real video sequence and a forged video sequence; A data construction unit is used to select several video clips from the real video sequence as reference samples, randomly select several video clips from the real video sequence as positive samples, select several video clips from the forged video sequence as negative samples, and construct triplet data based on the reference samples, positive samples and negative samples.

6. A computer terminal device, characterized in that: include: one or more processors; a memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the forged video detection method based on latent space learning according to any one of claims 1 to 3.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the forged video detection method based on latent space learning according to any one of claims 1 to 3 is implemented.

Images