A method, apparatus and equipment for detecting fake videos
By constructing and training a model architecture, utilizing deep separable convolutional networks and multilayer perceptual fully connected neural networks, combined with the fast Fourier transform algorithm and cross-entropy loss function, the problem of fake video recognition and algorithm tracing was solved, achieving efficient and accurate fake video detection and algorithm tracing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NAT UNIV OF DEFENSE TECH
- Filing Date
- 2023-10-13
- Publication Date
- 2026-07-17
AI Technical Summary
Existing technologies are insufficient to accurately and efficiently identify fake videos and trace their generation algorithms, making it difficult to control the harmfulness of fake videos.
A model architecture was constructed, including a spatial defect extraction module, a dynamic defect extraction module, and a classification module. The detection model was trained using a deep separable convolutional network and a multilayer perceptual fully connected neural network, combined with a fast Fourier transform algorithm and a cross-entropy loss function, to identify fake videos and determine the generation algorithm.
It achieves accurate detection of fake videos, significantly improves detection efficiency, and supports the tracing of fake video generation algorithms, reducing their harmfulness.
Smart Images
Figure CN117649621B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer technology, and specifically relates to a method, apparatus and equipment for detecting fake videos. Background Technology
[0002] The deepfake technology that has emerged in recent years has truly shattered people's traditional perception of "seeing is believing." The misuse of this technology can have irreparable and serious consequences for user security, and it must be taken very seriously.
[0003] Generally, videos contain more and richer information, making a fake video more credible than a fake image. Forging videos and images can be easily accomplished using open-source software, allowing even ordinary users without any professional knowledge to easily create them at a low cost.
[0004] Because the dynamic information contained in videos greatly enhances public credibility, fake videos pose a greater potential threat from a practical application perspective, making the research into effective deepfake video detection technologies urgent. Besides determining the authenticity of a video, tracing the algorithm used to generate it is also crucial. This source tracing capability can, to some extent, reduce the spread and credibility of deepfake videos, further mitigating their harmfulness. Summary of the Invention
[0005] The technical problem to be solved by the present invention is to provide a method, apparatus and equipment for detecting fake videos that can accurately and efficiently identify fake videos.
[0006] The present invention includes providing a method for detecting fake videos, comprising:
[0007] The obtained video dataset containing facial features is processed to form an image sequence set, which contains a continuous sequence of facial images corresponding to each video data. The video dataset includes fake video data and real video data.
[0008] A model architecture is constructed, comprising a spatial defect extraction module for extracting features of spatial artifacts and tampering traces in each frame of image, a dynamic defect extraction module for extracting dynamic defect features from the features extracted by the spatial defect extraction module, and a classification module for classifying and detecting the features extracted by the dynamic defect extraction module to at least determine whether the input video is a fake video.
[0009] The model architecture is trained based on the image sequence set and the first loss function and the second loss function to obtain a detection model that can detect fake videos and / or determine the fake algorithm used. The first loss function is used to train the results output by the classification module, and the second loss function is used to train the data output by the dynamic defect extraction module.
[0010] Obtain the video data to be tested, and process the video data to be tested to form a set of image sequences to be tested;
[0011] The set of images to be tested is input into the detection model to obtain the corresponding detection results.
[0012] In some embodiments, processing the obtained video dataset containing facial features to form an image sequence set includes:
[0013] The obtained video data containing facial features is subjected to image frame extraction to obtain multiple consecutive frame images corresponding to the video data;
[0014] Face recognition is performed on each frame image, and a face image of the target size is obtained by cropping. Multiple face images generated from the consecutive frame images form the face image sequence, and the face image sequences of all the video data form the image sequence set.
[0015] In some embodiments, the method further includes:
[0016] The spatial defect extraction module is constructed based on a depthwise separable convolutional network.
[0017] The image sequence set is input to the spatial defect extraction module, which then performs the following processing:
[0018] Feature extraction is performed on each frame of the image sequence to form a feature map;
[0019] The frequency domain sensing features of each feature map are calculated based on the Fast Fourier Transform algorithm to obtain the frequency domain sensing features of each feature map, and the deep feature map of each frame image is obtained by processing the obtained frequency domain sensing features.
[0020] In some embodiments, the method further includes:
[0021] Spatial average pooling is performed on each of the aforementioned feature maps and deep feature maps to obtain a first feature map set and a first deep feature map set.
[0022] In some embodiments, the method further includes:
[0023] The first feature map set is input into the dynamic defect extraction module, which then performs the following processing:
[0024] The first feature map is processed to obtain a first feature sequence corresponding to each video data, wherein the dimension of the first feature sequence is the first dimension;
[0025] A first representative feature for learning is added at the first position of each of the first feature sequences;
[0026] A first embedding feature is added at the second position of each of the first feature sequences to preserve the position information of the corresponding first feature sequence;
[0027] The first feature sequence, after the features have been added, is processed by an encoder to obtain the first target feature vector.
[0028] In some embodiments, the method further includes:
[0029] The first deep feature map set is input into the dynamic defect extraction module, which then performs the following processing:
[0030] The first deep feature map is processed to obtain a second feature sequence corresponding to each video data, and the dimension of the second feature sequence is the second dimension;
[0031] A second representative feature for learning is added at the first position of each second feature sequence;
[0032] A second embedding feature is added at the second position of each second feature sequence to preserve the position information of the corresponding second feature sequence;
[0033] The second feature sequence, after the features have been added, is processed by the encoder to obtain the second target feature vector.
[0034] In some embodiments, the method further includes:
[0035] The classification module is constructed based on a multilayer perceptron fully connected neural network;
[0036] The first target feature vector and the second target feature vector are input into the classification module to obtain the first detection result and the second detection result;
[0037] The average of the first and second test results is taken as the final test result.
[0038] In some embodiments, the first loss function is a cross-entropy loss function, and the classification module is also capable of performing a detection task to determine the forgery algorithm used in the forged video;
[0039] The training of the model architecture based on the image sequence set and the first and second loss functions includes:
[0040] The first label representing real video and the second label representing fake video or fake video generation algorithm are assigned values based on the detection task.
[0041] Construct corresponding hard sample mining loss functions for the first target feature vector and the second target feature vector respectively;
[0042] The model architecture is trained based on the assigned values, image sequence set, cross-entropy loss function, and hard sample mining loss functions corresponding to the first target feature vector and the second target feature vector, respectively.
[0043] Another embodiment of the present invention also provides a fake video detection device, comprising:
[0044] The processing module is used to process the obtained video dataset containing facial features to form an image sequence set. The image sequence set contains a continuous facial image sequence corresponding to each video dataset. The video dataset includes fake video data and real video data.
[0045] A construction module is used to construct a model architecture, which includes a spatial defect extraction module for extracting features of spatial artifacts and tampering traces of each frame image, a dynamic defect extraction module for extracting dynamic defect features from the features extracted by the spatial defect extraction module, and a classification module for classifying and detecting the features extracted by the dynamic defect extraction module to at least determine whether the input video is a fake video.
[0046] The training module is used to train the model architecture based on the image sequence set and the first loss function and the second loss function to obtain a detection model that can detect fake videos and / or determine the fake algorithm used. The first loss function is used to train the results output by the classification module, and the second loss function is used to train the data output by the dynamic defect extraction module.
[0047] The acquisition module is used to acquire the video data to be tested, and to process the video data to be tested based on the processing module to form a set of image sequences to be tested;
[0048] The input module is used to input the set of images to be tested into the detection model to obtain the corresponding detection results.
[0049] Another embodiment of the present invention also provides an electronic device, comprising:
[0050] At least one processor; and,
[0051] A memory communicatively connected to the at least one processor; wherein,
[0052] The memory stores instructions that can be executed by the at least one processor to implement the fake video detection method as described in any one of the above descriptions.
[0053] The beneficial effect of this invention is that it constructs a model architecture and trains it to more accurately detect forged videos by utilizing temporal discontinuities and combining them with spatial defects. This not only achieves accurate detection of forged videos but also significantly improves detection efficiency. Simultaneously, the trained detection model can also determine the number of algorithms involved in forged videos, providing a reference for algorithm attribution. Attached Figure Description
[0054] Figure 1 This is a flowchart of the fake video detection method of the present invention.
[0055] Figure 2 This is a flowchart illustrating the application of the fake video detection method of the present invention.
[0056] Figure 3 This is a structural block diagram of the fake video detection device of the present invention. Detailed Implementation
[0057] like Figure 1 As shown, the present invention includes a method for detecting fake videos, comprising:
[0058] S1: The obtained video dataset containing facial features is processed to form an image sequence set. The image sequence set contains a continuous sequence of facial images corresponding to each video data. The video dataset includes fake video data and real video data.
[0059] S2: Construct the model architecture, which includes a spatial defect extraction module for extracting features of spatial artifacts and tampering traces in each frame of image, a dynamic defect extraction module for extracting dynamic defect features from the features extracted by the spatial defect extraction module, and a classification module for classifying and detecting the features extracted by the dynamic defect extraction module to at least determine whether the input video is a fake video.
[0060] S3: The model architecture is trained based on the image sequence set and the first loss function and the second loss function to obtain a detection model that can detect fake videos and / or determine the fake algorithm used. The first loss function is used to train the results output by the classification module, and the second loss function is used to train the data output by the dynamic defect extraction module.
[0061] S4: Obtain the video data to be tested, and process the video data to be tested to form a set of image sequences to be tested;
[0062] S5: Input the set of image sequences to be tested into the detection model to obtain the corresponding detection results.
[0063] The beneficial effect of this embodiment is that it constructs a model architecture and trains it to more accurately detect forged videos by utilizing temporal discontinuities and combining them with relevant defects in the spatial domain. This not only achieves accurate detection of forged videos but also significantly improves detection efficiency. Simultaneously, the trained detection model can also support determining the number of algorithms involved in forged videos, providing a reference for algorithm attribution.
[0064] Specifically, when processing the obtained video dataset containing facial features to form an image sequence set, the process includes:
[0065] The obtained video data containing facial features is processed to extract image frames, resulting in multiple consecutive frame images of the corresponding video data;
[0066] Face recognition is performed on each frame image, and a face image of the target size is cropped. Multiple face images generated from consecutive frame images form a face image sequence, and the face image sequences of all video data form an image sequence set.
[0067] For example, firstly, extract N consecutive frames from the video data, or divide the video data into N consecutive frames. Then, use face recognition tools such as Dlib to detect face regions in each frame. Afterward, crop out the detected face regions, and uniformly adjust the cropped face images to 224×224 pixels. The sequence of N consecutive face images obtained through the above processing is denoted as X = [X1, X2, ..., X...]. N The sequence of facial images from all video data forms an image sequence set.
[0068] Furthermore, when constructing the model architecture, multiple modules need to be built sequentially. The following explains the construction of each module and the data processing process of each module:
[0069] S6: Construct a spatial defect extraction module based on a depthwise separable convolutional network;
[0070] The image sequence set is input into the spatial defect extraction module, which then performs the following processing:
[0071] S7: Extract features from each frame of the image sequence to form a feature map;
[0072] S8: Calculate the frequency domain sensing features of each feature map based on the Fast Fourier Transform algorithm, and process the obtained frequency domain sensing features to obtain the deep feature map of each frame image.
[0073] Specifically, such as Figure 2 As shown, the spatial defect extraction module in this embodiment uses XceptionNet (Depthally Separable Convolutional Network) as its backbone network to extract spatial artifacts and tampering traces from each frame of X (the aforementioned face image sequence). The XceptionNet network consists of four parts: Stem (convolutional) layers, Entry flow (input layer), Middle flow (intermediate layer), and deep Exit flow (output layer). The features of X output by the Entry flow are represented as... Where N, c1, h1, and w1 represent the number of frames in the video sequence, the number of channels in the Entry flow output feature map, the height, and the width, respectively. For the feature map corresponding to the face image in the i-th frame (i∈[1,N]), the spatial defect extraction module uses Fast Fourier Transform (FFT) to obtain the spectral representation, denoted as […]. To make the frequency domain response more flexible, this embodiment will use F i With an additional learnable filter Perform a dot product to adaptively adjust the frequency domain response, i.e.: G' i =G i ⊙F i .
[0074] Finally, the inverse fast Fourier transform (iFFT) is used to transform G' i Transforming back to the spatial domain yields frequency domain sensing features. Through the above process, the feature maps of N frames will eventually generate frequency domain-aware features. This feature can supplement the RGB domain information in an image to uncover more subtle clues of forgery. Then... The data is then fed into the Middle flow layer of the backbone network for further processing, and then into the Exit flow for further processing to generate deep feature maps of the corresponding video data. The deep feature map output by the Exit flow is denoted as... Where N, c2, h2, and w2 represent the number of frames in the video sequence, the number of channels in the feature map M2, its height, and its width, respectively.
[0075] To reduce data dimensionality and avoid overfitting during model training, the method in this embodiment further includes:
[0076] S9: Perform spatial average pooling on each feature map and deep feature map to obtain the first feature map set and the first deep feature map set.
[0077] For example, the intermediate layer feature map output by Entry flow in the above steps Spatial average pooling is used for processing to obtain features. The deep feature map output by Exit flow Spatial average pooling is used for processing to obtain features.
[0078] Furthermore, a dynamic defect extraction module is constructed. Subsequently, the method in this embodiment further includes:
[0079] The first feature map set is input into the dynamic defect extraction module, which then performs the following processing:
[0080] S10: Process the first feature map set to obtain the first feature sequence corresponding to each video data. The dimension of the first feature sequence is the first dimension.
[0081] S11: Add the first representative feature for learning at the first position of each first feature sequence;
[0082] S12: Add a first embedding feature at the second position of each first feature sequence to retain the position information of the corresponding first feature sequence;
[0083] S13: The first feature sequence after adding features is processed by the encoder to obtain the first target feature vector.
[0084] Specifically, in M1' of this embodiment, the feature vector corresponding to each frame of image has a size of c1×1×1. After flattening, N consecutive frames corresponding to a video data will form a 1D embedding sequence of length N, represented as follows: Where t∈[1,N]. Subsequently, a trainable linear projection layer E is used to change the dimension of each embedding feature sequence from c1 to D. To achieve classification, the dynamic defect extraction module in this embodiment adds an additional learnable class embedding, denoted as f, before all embedding feature sequences. class This can serve as a representative feature to be learned on the input sequence. Furthermore, in this embodiment, a learnable E is embedded at the 1D position in the feature sequence. pos This is used to preserve location information. Ultimately, the resulting embedding sequence is represented as...
[0085] Next, h0 is fed into the Transformer Encoder module, and the module output f is... classThe feature vector at the location is denoted as z1, which is the first target feature vector.
[0086] The first deep feature map set is processed using the same method described above, as follows:
[0087] S14: Input the first deep feature map set into the dynamic defect extraction module, which then performs the following processing:
[0088] S15: Process the first deep feature map to obtain the second feature sequence corresponding to each video data. The dimension of the second feature sequence is the second dimension.
[0089] S16: Add a second representative feature for learning at the first position of each second feature sequence;
[0090] S17: Add a second embedding feature at the second position of each second feature sequence to preserve the position information of the corresponding second feature sequence;
[0091] S18: The second feature sequence with added features is processed by the encoder to obtain the second target feature vector.
[0092] Furthermore, the specific steps for constructing the classification module are as follows:
[0093] S19: Construct a classification module based on a multilayer perceptual fully connected neural network (MLP network);
[0094] S20: Input the first target feature vector and the second target feature vector into the classification module to obtain the first detection result and the second detection result;
[0095] S21: Take the average of the first and second test results as the final test result.
[0096] The purpose of averaging the two detection results as the final model output is to improve the model's detection accuracy.
[0097] After constructing the aforementioned modules, the model needs to be trained. The data processing procedures performed by each module after obtaining the data to be processed constitute the data processing flow during training and practical application. To ensure sufficient training and higher accuracy of the resulting detection model, this embodiment employs two different loss functions for training different feature data during model architecture training. Furthermore, the classification module in this embodiment can also perform the detection task of determining the forgery algorithm used in the forged video. When requiring the classification module or model to perform different detection tasks, different parameters need to be set for the labels in the classification module to ensure that the output label content matches the detection task. Specifically:
[0098] In this embodiment, the first loss function is the cross-entropy loss function (CE Loss). When training the model architecture based on the image sequence set and the first and second loss functions, the following steps are included:
[0099] S22: Assign values to the first label representing the real video and the second label representing the fake video or the fake video generation algorithm based on the detection task;
[0100] S23: Construct corresponding hard sample mining loss functions for the first target feature vector and the second target feature vector respectively;
[0101] S24: The model architecture is trained based on assignment, image sequence set, cross-entropy loss function and hard sample mining loss function corresponding to the first target feature vector and the second target feature vector respectively.
[0102] For example, the cross-entropy loss function can be expressed as:
[0103] Where N is the total number of training video samples, y i and These are the predicted label / detected label and the real label for the i-th sample, respectively. For fake video detection tasks, the real video's... Set to 0, y of the fake video i Set to 1. For tasks involving tracing the origins of fake algorithms, the real video will be... Setting it to 0, the y-value of the fake videos generated by different fakery algorithms i They are set to 1, 2, 3... respectively, depending on the number of forgery algorithms.
[0104] Next, continue to combine Figure 2 As shown, hard sample mining loss functions are constructed for the first target feature vector z1 and the second target feature vector z2, respectively. Taking z1 as an example, firstly, all its triplet combinations are found in the same training batch, namely anchor, positive sample, and negative sample. Then, for each triplet...<a,p,n> Calculate r = (ap) - (an), which is the difference between the anchor and the positive sample minus the difference between the anchor and the negative sample. Clearly, the larger r is, the harder it is for the model to make a correct classification, meaning the detection error is larger. After calculating r for all triplet combinations, sort all the obtained r from largest to smallest, and select the top K corresponding triplet combinations as the hard sample mining targets. Next, use SoftMargin Loss as the hard sample mining loss function, and define the loss function corresponding to the first target feature vector as:
[0105]
[0106] In the above formula, x[i] represents the r value calculated for the i-th triplet, and y i ∈{1}. Similarly, the hard sample mining loss function obtained by calculating the feature vector z2 in the above way is expressed as L. hard_2 Finally, the three loss functions mentioned above are integrated to form the loss function corresponding to the detection model in this embodiment, and training is performed based on this loss function. The specific loss function in this embodiment is as follows:
[0107] L total =L ce +L hard_1 +L hard_2
[0108] Once the loss function is obtained, the model architecture is complete, and the training data can then be used to train the model. The trained model can then perform authenticity detection or fake video generation algorithm detection on new input videos, i.e., the test videos.
[0109] As mentioned above, after the detection model receives input data, it undergoes corresponding processing through the various modules described above until the classification module outputs the final detection result. The model architecture in this embodiment is not an existing model architecture, but rather a matching construction based on existing models with different functions, aimed at solving the technical problems of this embodiment. Each module in the model processes data based on its basic functions and is implemented according to the methods designed in this embodiment. Furthermore, the preparation of training data is also based on the methods designed in this embodiment. The detection model constructed and trained based on the methods of this embodiment can accurately and efficiently detect forged videos or forged video generation algorithms, providing a guarantee for protecting user information security.
[0110] like Figure 3 As shown, another embodiment of the present invention also provides a fake video detection device 100, comprising:
[0111] The processing module is used to process the obtained video dataset containing facial features to form an image sequence set. The image sequence set contains a continuous facial image sequence corresponding to each video dataset. The video dataset includes fake video data and real video data.
[0112] A construction module is used to construct a model architecture, which includes a spatial defect extraction module for extracting features of spatial artifacts and tampering traces of each frame image, a dynamic defect extraction module for extracting dynamic defect features from the features extracted by the spatial defect extraction module, and a classification module for classifying and detecting the features extracted by the dynamic defect extraction module to at least determine whether the input video is a fake video.
[0113] The training module is used to train the model architecture based on the image sequence set and the first loss function and the second loss function to obtain a detection model that can detect fake videos and / or determine the fake algorithm used. The first loss function is used to train the results output by the classification module, and the second loss function is used to train the data output by the dynamic defect extraction module.
[0114] The acquisition module is used to acquire the video data to be tested, and to process the video data to be tested based on the processing module to form a set of image sequences to be tested;
[0115] The first input module is used to input the set of image sequences to be tested into the detection model to obtain the corresponding detection results.
[0116] In some embodiments, processing the obtained video dataset containing facial features to form an image sequence set includes:
[0117] The obtained video data containing facial features is subjected to image frame extraction to obtain multiple consecutive frame images corresponding to the video data;
[0118] Face recognition is performed on each frame image, and a face image of the target size is obtained by cropping. Multiple face images generated from the consecutive frame images form the face image sequence, and the face image sequences of all the video data form the image sequence set.
[0119] In some embodiments, the building module is further configured to:
[0120] The spatial defect extraction module is constructed based on a depthwise separable convolutional network.
[0121] The image sequence set is input to the spatial defect extraction module, which then performs the following processing:
[0122] Feature extraction is performed on each frame of the image sequence to form a feature map;
[0123] The frequency domain sensing features of each feature map are calculated based on the Fast Fourier Transform algorithm to obtain the frequency domain sensing features of each feature map, and the deep feature map of each frame image is obtained by processing the obtained frequency domain sensing features.
[0124] In some embodiments, the apparatus further includes:
[0125] The pooling module is used to perform spatial average pooling operations on each of the feature maps and deep feature maps to obtain a first feature map set and a first deep feature map set.
[0126] In some embodiments, the apparatus further includes:
[0127] The second input module is used to input the first feature map set into the dynamic defect extraction module, whereby the dynamic defect extraction module performs the following processing:
[0128] The first feature map is processed to obtain a first feature sequence corresponding to each video data, wherein the dimension of the first feature sequence is the first dimension;
[0129] A first representative feature for learning is added at the first position of each of the first feature sequences;
[0130] A first embedding feature is added at the second position of each of the first feature sequences to preserve the position information of the corresponding first feature sequence;
[0131] The first feature sequence, after the features have been added, is processed by an encoder to obtain the first target feature vector;
[0132] In some embodiments, the apparatus further includes:
[0133] The third input module is used to input the first deep feature map set into the dynamic defect extraction module, which then performs the following processing:
[0134] The first deep feature map is processed to obtain a second feature sequence corresponding to each video data, and the dimension of the second feature sequence is the second dimension;
[0135] A second representative feature for learning is added at the first position of each second feature sequence;
[0136] A second embedding feature is added at the second position of each second feature sequence to preserve the position information of the corresponding second feature sequence;
[0137] The second feature sequence, after the features have been added, is processed by an encoder to obtain the second target feature vector;
[0138] In some embodiments, the building module is further configured to:
[0139] The classification module is constructed based on a multilayer perceptron fully connected neural network;
[0140] The first target feature vector and the second target feature vector are input into the classification module to obtain the first detection result and the second detection result;
[0141] The average of the first and second test results is taken as the final test result.
[0142] In some embodiments, the first loss function is a cross-entropy loss function, and the classification module is also capable of performing a detection task to determine the forgery algorithm used in the forged video;
[0143] The training of the model architecture based on the image sequence set and the first and second loss functions includes:
[0144] The first label representing real video and the second label representing fake video or fake video generation algorithm are assigned values based on the detection task.
[0145] Construct corresponding hard sample mining loss functions for the first target feature vector and the second target feature vector respectively;
[0146] The model architecture is trained based on the assigned values, image sequence set, cross-entropy loss function, and hard sample mining loss functions corresponding to the first target feature vector and the second target feature vector, respectively.
[0147] Another embodiment of the present invention also provides an electronic device, comprising:
[0148] At least one processor; and,
[0149] A memory communicatively connected to the at least one processor; wherein,
[0150] The memory stores instructions that can be executed by the at least one processor to implement the fake video detection method as described in any of the embodiments above.
[0151] Furthermore, one embodiment of the present invention also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the forged video detection method described above. It should be understood that the various solutions in this embodiment have the corresponding technical effects in the above-described method embodiments, and will not be repeated here.
[0152] Furthermore, embodiments of the present invention also provide a computer program product, which is tangibly stored on a computer-readable medium and includes computer-readable instructions that, when executed, cause at least one processor to perform a fake video detection method such as those described in the embodiments above.
[0153] It should be noted that the computer storage medium of the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access storage medium (RAM), a read-only storage medium (ROM), an erasable programmable read-only storage medium (EPROM or flash memory), an optical fiber, a portable compact disk read-only storage medium (CD-ROM), an optical storage medium, a magnetic storage medium, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program configured for use by or in connection with an instruction execution system, system, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, antenna, optical fiber, RF, etc., or any suitable combination thereof.
[0154] Furthermore, those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.
[0155] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.
[0156] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0157] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0158] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of protection of this application is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in detail for the sake of brevity.
[0159] One or more embodiments in this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments in this application should be included within the protection scope of this application.
Claims
1. A method for detecting forged videos, characterized in that, include: The obtained video dataset containing facial features is processed to form an image sequence set, which contains a continuous sequence of facial images corresponding to each video data. The video dataset includes fake video data and real video data. A model architecture is constructed, comprising a spatial defect extraction module for extracting features of spatial artifacts and tampering traces in each frame of image, a dynamic defect extraction module for extracting dynamic defect features from the features extracted by the spatial defect extraction module, and a classification module for classifying and detecting the features extracted by the dynamic defect extraction module to at least determine whether the input video is a fake video. The model architecture is trained based on the image sequence set and the first loss function and the second loss function to obtain a detection model that can detect fake videos and / or determine the fake algorithm used. The first loss function is used to train the results output by the classification module, and the second loss function is used to train the data output by the dynamic defect extraction module. Obtain the video data to be tested, and process the video data to be tested to form a set of image sequences to be tested; The set of images to be tested is input into the detection model to obtain the corresponding detection results; The method further includes: The spatial defect extraction module is constructed based on a depthwise separable convolutional network. The image sequence set is input to the spatial defect extraction module, which then performs the following processing: Feature extraction is performed on each frame of the image sequence to form a feature map; The frequency domain sensing features of each feature map are calculated based on the Fast Fourier Transform algorithm to obtain the frequency domain sensing features of each feature map, and the deep feature map of each frame image is obtained by processing the obtained frequency domain sensing features. The method further includes: Spatial average pooling is performed on each of the aforementioned feature maps and deep feature maps to obtain a first feature map set and a first deep feature map set; The method further includes: The first feature map set is input into the dynamic defect extraction module, which then performs the following processing: The first feature map is processed to obtain a first feature sequence corresponding to each video data, wherein the dimension of the first feature sequence is the first dimension; A first representative feature for learning is added at the first position of each of the first feature sequences; A first embedding feature is added at the second position of each of the first feature sequences to preserve the position information of the corresponding first feature sequence; The first feature sequence, after the features have been added, is processed by an encoder to obtain the first target feature vector; The method further includes: The first deep feature map set is input into the dynamic defect extraction module, which then performs the following processing: The first deep feature map is processed to obtain a second feature sequence corresponding to each video data, and the dimension of the second feature sequence is the second dimension; A second representative feature for learning is added at the first position of each second feature sequence; A second embedding feature is added at the second position of each second feature sequence to preserve the position information of the corresponding second feature sequence; The second feature sequence, after the features have been added, is processed by the encoder to obtain the second target feature vector.
2. The method for detecting fake videos according to claim 1, characterized in that, The process of processing the obtained video dataset containing facial features to form an image sequence set includes: The obtained video data containing facial features is subjected to image frame extraction to obtain multiple consecutive frame images corresponding to the video data; Face recognition is performed on each frame image, and a face image of the target size is obtained by cropping. Multiple face images generated from the consecutive frame images form the face image sequence, and the face image sequences of all the video data form the image sequence set.
3. The method for detecting fake videos according to claim 1, characterized in that, The method further includes: The classification module is constructed based on a multilayer perceptron fully connected neural network; The first target feature vector and the second target feature vector are input into the classification module to obtain the first detection result and the second detection result; The average of the first and second test results is taken as the final test result.
4. The method for detecting fake videos according to claim 3, characterized in that, The first loss function is the cross-entropy loss function, and the classification module can also perform a detection task to determine the forgery algorithm used in the forged video; The training of the model architecture based on the image sequence set and the first and second loss functions includes: The first label representing real video and the second label representing fake video or fake video generation algorithm are assigned values based on the detection task. Construct corresponding hard sample mining loss functions for the first target feature vector and the second target feature vector respectively; The model architecture is trained based on the assigned values, image sequence set, cross-entropy loss function, and hard sample mining loss functions corresponding to the first target feature vector and the second target feature vector, respectively.
5. A device for detecting forged videos, characterized in that, Used to implement the fake video detection method as described in any one of claims 1-4.
6. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor to implement the fake video detection method as described in any one of claims 1-4.