Deepfake video detection method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a deep fake video detection method and device, electronic equipment and storage medium, relating to the technical field of computer. The method comprises: inputting an image set into a first target network model to obtain first features, and inputting the image set into a second target network model to obtain second features; performing time consistency analysis according to the first features to obtain third features, and performing light response non-uniformity analysis according to the second features to obtain fourth features; determining a first discrimination result according to the third features, and determining a second discrimination result according to the fourth features; and determining a video detection result according to the first discrimination result and the second discrimination result. According to the present disclosure, the first target network model and the second target network model are used for feature extraction, so that the features can be fully extracted. Moreover, the time consistency analysis and the light response non-uniformity analysis are introduced, so that the deep fake videos can be detected, the accuracy and reliability of the deep fake video detection are improved, and the network security is improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus, electronic device, and storage medium for detecting deepfake videos. Background Technology

[0002] With the rapid development of deepfake technology, the generated deepfake videos have reached a level that is difficult for the human eye to detect. This has led to an exponential increase in issues such as impersonating public figures, defamation, and the spread of false information, necessitating the development of effective deepfake detection methods to address the threat of false content.

[0003] However, because deepfake videos contain temporal features across different frames, and each frame contains subtle and imperceptible modifications, the frame-level visual effects become even more realistic. Currently, there is an urgent need for a detection method that can detect various types of deepfake videos to improve the accuracy and reliability of deepfake video detection, thereby enhancing cybersecurity.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] This disclosure provides a method, apparatus, electronic device, and storage medium for detecting deepfake videos, which at least to some extent overcomes the problems that related technologies cannot detect a variety of deepfake videos, have low accuracy and reliability in deepfake video detection, and cannot guarantee network security.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] According to one aspect of the present disclosure, a method for detecting deepfake videos is provided, comprising:

[0008] The first feature is obtained by inputting the image set consisting of each frame of the video to be detected into the first target network model, and the second feature is obtained by inputting the image set into the second target network model. The first target network model and the second target network model are used for feature extraction based on different algorithms.

[0009] Based on the first feature, a time consistency analysis is performed to obtain the third feature, and based on the second feature, an optical response non-uniformity analysis is performed to obtain the fourth feature.

[0010] The first discrimination result is determined based on the third feature, and the second discrimination result is determined based on the fourth feature;

[0011] The video detection result is determined based on the first discrimination result and the second discrimination result.

[0012] In some embodiments of this disclosure, a third feature is obtained by performing time consistency analysis based on the first feature, including:

[0013] The first feature is input into the BiLSTM (Bi-directional Long Short-Term Memory) model to obtain the third feature.

[0014] In some embodiments of this disclosure, a fourth feature is obtained by performing photoresponse nonuniformity analysis based on the second feature, including:

[0015] The second feature is input into the PRNU (Photo-Response Non-Uniformity) model to obtain the fourth feature.

[0016] In some embodiments of this disclosure, the first target network model is obtained by fusing two CNN (Convolutional Neural Networks) models with different activation functions.

[0017] In some embodiments of this disclosure, the first target network model is obtained by fusing an InceptionResNet model and an Efficient-bo model, wherein the activation function corresponding to the InceptionResNet model is swish, and the activation function corresponding to the Efficient-bo model is ReLU (Rectified Linear Unit).

[0018] In some embodiments of this disclosure, determining the first discrimination result based on the third feature includes:

[0019] The third feature is input into the SVR (Support Vector Regression) model to obtain the first discrimination result;

[0020] The SVR model includes multiple preset non-sensitive region widths, each non-sensitive region width corresponds to a first discrimination value, and the first discrimination result is determined based on multiple first discrimination values.

[0021] In some embodiments of this disclosure, determining a video detection result based on the first discrimination result and the second discrimination result includes:

[0022] Determine the first evaluation index corresponding to the first discrimination result and the second evaluation index corresponding to the second discrimination result;

[0023] The fusion ratio is obtained by fitting the first evaluation index and the second evaluation index to determine the video detection result.

[0024] In some embodiments of this disclosure, before inputting the image set composed of each frame of the video to be detected into the first target network model to obtain the first feature, the deepfake video detection method provided in the embodiments of this disclosure may further include:

[0025] Obtain the video to be detected; perform frame extraction, face detection, and grayscale conversion on the video to obtain the image set.

[0026] According to another aspect of this disclosure, a deepfake video detection apparatus is provided, comprising:

[0027] The feature extraction module is used to input an image set consisting of each frame of the video to be detected into a first target network model to obtain a first feature, and input the image set into a second target network model to obtain a second feature. The first target network model and the second target network model are used to perform feature extraction based on different algorithms.

[0028] The feature analysis module is used to perform time consistency analysis based on the first feature to obtain the third feature, and to perform optical response non-uniformity analysis based on the second feature to obtain the fourth feature.

[0029] The discrimination result determination module is used to determine a first discrimination result based on the third feature and a second discrimination result based on the fourth feature;

[0030] The video detection result determination module is used to determine the video detection result based on the first discrimination result and the second discrimination result.

[0031] In some embodiments of this disclosure, a feature analysis module is used to input the first feature into a BiLSTM model to obtain the third feature.

[0032] In some embodiments of this disclosure, a feature analysis module is used to input the second feature into the PRNU model to obtain the fourth feature.

[0033] In some embodiments of this disclosure, the first target network model is obtained by fusing two CNN models with different activation functions.

[0034] In some embodiments of this disclosure, the first target network model is obtained by fusing the InceptionResNet model and the Efficient-bo model, wherein the activation function corresponding to the InceptionResNet model is swish and the activation function corresponding to the Efficient-bo model is ReLU.

[0035] In some embodiments of this disclosure, the discrimination result determination module is used to input the third feature into the SVR model to obtain the first discrimination result; wherein, the SVR model includes a plurality of preset non-sensitive region widths, each non-sensitive region width corresponds to a first discrimination value, and the first discrimination result is determined based on the plurality of first discrimination values.

[0036] In some embodiments of this disclosure, a video detection result determination module is used to determine a first evaluation index corresponding to the first discrimination result and a second evaluation index corresponding to the second discrimination result; and to determine the video detection result by fitting a fusion ratio using the first evaluation index and the second evaluation index.

[0037] In some embodiments of this disclosure, the deepfake video detection apparatus provided in this disclosure further includes:

[0038] The image set determination module is used to acquire the video to be detected; and to perform frame extraction, face detection, and grayscale conversion on the video to be detected to obtain the image set.

[0039] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the aforementioned deepfake video detection method by executing the executable instructions.

[0040] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the above-described deepfake video detection method.

[0041] According to another aspect of this disclosure, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the deepfake video detection method provided in various alternative embodiments of this disclosure.

[0042] The technical solution provided in this disclosure can perform feature extraction by jointly using a first target network model and a second target network model, which can improve the sufficiency of feature extraction and retain more feature information. Furthermore, this disclosure introduces temporal consistency analysis and optical response non-uniformity analysis, thereby enabling the detection of various deepfake videos, improving the accuracy and reliability of deepfake video detection, and thus enhancing network security.

[0043] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0045] Figure 1 A schematic diagram of a system architecture according to an embodiment of this disclosure is shown;

[0046] Figure 2 This diagram illustrates a flowchart of a deepfake video detection method according to an embodiment of the present disclosure;

[0047] Figure 3 A schematic diagram of a hybrid deep learning framework according to an embodiment of this disclosure is shown;

[0048] Figure 4 A flowchart illustrating a deepfake video detection method according to an embodiment of this disclosure is shown;

[0049] Figure 5 This diagram illustrates a deepfake video detection device according to an embodiment of the present disclosure.

[0050] Figure 6 This diagram illustrates a structural block diagram of an electronic device according to an embodiment of the present disclosure.

[0051] Figure 7 A schematic diagram of a computer-readable storage medium according to an embodiment of the present disclosure is shown. Detailed Implementation

[0052] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0053] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0054] Figure 1 A schematic diagram of an exemplary system architecture that can be applied to the deepfake video detection method of the embodiments of this disclosure is shown.

[0055] like Figure 1 As shown, the system architecture provided in this embodiment may include a terminal device 101, a network 102, and a server 103.

[0056] The terminal device 101 can send the video to be detected to the server 103.

[0057] Then, the server 103 can input the image set composed of each frame image corresponding to the video to be detected into the first target network model to obtain the first feature, and input the image set into the second target network model to obtain the second feature. The first target network model and the second target network model are used for feature extraction based on different algorithms.

[0058] Then, server 103 can perform time consistency analysis based on the first feature to obtain a third feature, and perform optical response non-uniformity analysis based on the second feature to obtain a fourth feature. Furthermore, it can determine a first discrimination result based on the third feature and a second discrimination result based on the fourth feature.

[0059] Finally, server 103 can determine the video detection result based on the first discrimination result and the second discrimination result, and send the video detection result to terminal device 101.

[0060] Network 102 is a medium used to provide a communication link between terminal device 101 and server 103, and can be a wired network or a wireless network.

[0061] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to Local Area Networks (LANs), Metropolitan Area Networks (MANs), Wide Area Networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0062] Terminal device 101 can be various electronic devices, including but not limited to smartphones, tablets, laptops, desktop computers, wearable devices, augmented reality devices, virtual reality devices, etc.

[0063] Server 103 can be a server that provides various services, such as a backend management server that supports the device operated by the user using terminal device 101. The backend management server can analyze and process received requests and other data, and feed the processing results back to the terminal device.

[0064] Optionally, the server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0065] Those skilled in the art will know that Figure 1The number of terminal devices 101, networks 102, and servers 103 shown is merely illustrative. Any number of terminal devices 101, networks 102, and servers 103 can be used as needed. This disclosure does not limit the number of such devices.

[0066] The following detailed description of this exemplary implementation method is provided in conjunction with the accompanying drawings and embodiments.

[0067] First, this disclosure provides a method for detecting deepfake videos, which can be executed by any electronic device with computing power.

[0068] Figure 2 This diagram illustrates a flowchart of a deepfake video detection method according to an embodiment of the present disclosure, such as... Figure 2 As shown, the deepfake video detection method provided in this embodiment includes the following steps S202 to S208.

[0069] S202, the image set consisting of each frame of the video to be detected is input into the first target network model to obtain the first feature, and the image set is input into the second target network model to obtain the second feature. The first target network model and the second target network model are used for feature extraction based on different algorithms.

[0070] This disclosure does not limit the content of the video to be detected; the content can be determined based on the application scenario. For example, when the application scenario is identity verification using facial data, the content of the video to be detected can be a video containing faces. Alternatively, when the application scenario is pedestrian re-identification, the video to be detected can be a video containing at least one pedestrian.

[0071] For example, when the deepfake video involves the forgery of faces, face detection can be performed on each frame of the video to be detected.

[0072] In some embodiments, before inputting the image set composed of each frame image corresponding to the video to be detected into the first target network model to obtain the first feature, the deepfake video detection method provided in this disclosure may further include: acquiring the video to be detected; performing frame extraction, face detection, and grayscale conversion processing on the video to be detected to obtain the image set.

[0073] In one possible implementation, frames can be extracted from the video to be detected using OpenCV (Open Source Computer Vision Library). When each frame is a color image, it can be converted into a grayscale image for the detector to recognize.

[0074] Next, a Haar (a feature) detector (face_detector) can be constructed, and face detection can begin using the detectMultiScale (a function for object detection) method. After detection is complete, iterator parsing and parameter tuning can be performed. For example, the scaleFactor can be used to adjust the range of detected face sizes.

[0075] In some embodiments, the first target network model is obtained by fusing two CNN models with different activation functions. This disclosure does not limit the types of the two CNN models used here; any two CNN models suitable for feature extraction can be used.

[0076] It should be noted that selecting two CNN models with different activation functions for feature extraction here helps to fully extract features and retain more feature information.

[0077] In some exemplary embodiments, the first target network model is obtained by fusing the InceptionResNet model and the Efficient-bo model, wherein the activation function corresponding to the InceptionResNet model is swish and the activation function corresponding to the Efficient-bo model is ReLU.

[0078] For example, both the InceptionResNet model and the Efficient-bo model can be pre-trained for pre-parameter tuning. During the pre-training of the InceptionResNet model, the initial deep neural network architecture can be combined with a residual network, and the residual links can be mixed with convolutional filters of different sizes in each Inception-ResNet block. Furthermore, introducing a gradient-unsaturated ReLU activation function can accelerate convergence. In this case, the layers of the InceptionResNet model do not need to be frozen, thus achieving complete training of the InceptionResNet model.

[0079] In an exemplary embodiment, when pre-training the Efficient-bo model, the activation function in the Efficient-bo model can be set to swish, and then the initial E1 layer of the Efficient-bo model can be frozen. This allows for partial training on the constructed dataset while retaining ImageNet features (dataset features). In other words, only the part of the Efficient-bo model other than the initial E1 layer can be trained.

[0080] In some embodiments, after two CNN models are pre-trained, they can be concatenated and fused to integrate global average pooling layers, dense layers, etc., into the first target network model. Therefore, the embodiments of this disclosure can achieve feature analysis and deduction at a faster speed and with lower computational complexity.

[0081] For example, two CNN models can be fused together using the following formula (1).

[0082]

[0083] In formula (1), x can be the model input data, y1 can be the first feature, and β can be the parameter of the swish activation function. Specifically, x*sigmoid(βx) corresponds to the feature value obtained based on the swish activation function, and max(0, x) corresponds to the feature value obtained based on the ReLU activation function. Therefore, fusing the feature value obtained based on the swish activation function with the feature value obtained based on the ReLU activation function yields the first feature.

[0084] After obtaining the first target network model through cascading fusion, the acquired image set can be input into the first target network model to extract the first feature, thereby ensuring the sufficiency and reliability of feature extraction and further improving the accuracy and reliability of deepfake video detection.

[0085] In some embodiments, the image set can be input into a second target network model to obtain second features. The second target network model can be a ShuffleNet (a convolutional neural network model) model.

[0086] For example, pre-trained ImageNet weights can be used in the first layer of the ShuffleNet model, and global average pooling and dense layers can be added to the ShuffleNet model. After model tuning, the image set can be input into the ShuffleNet model to obtain the corresponding second features.

[0087] S204, based on the first feature, a time consistency analysis is performed to obtain the third feature, and based on the second feature, a light response non-uniformity analysis is performed to obtain the fourth feature.

[0088] In some embodiments, a BiLSTM model can be used as a back-end classifier to capture inconsistencies along the timeline for identifying temporal information. For example, eye blinks can have good correlations between nearby frames.

[0089] For example, performing time consistency analysis based on the first feature to obtain the third feature includes: inputting the first feature into a BiLSTM model to obtain the third feature.

[0090] In an exemplary embodiment, cross-entropy loss can be used as the loss function in the BiLSTM model, and two 128-unit LSTM layers can be constructed to enhance the feature learning ability, thereby ensuring the extraction capability of the third feature.

[0091] It should be noted that by performing time consistency analysis based on this first feature, the bidirectional propagation correlation of long-term dependencies can be extracted, which helps to improve the accuracy and reliability of deepfake video detection.

[0092] In some embodiments, performing photoresponse nonuniformity analysis based on the second feature to obtain the fourth feature includes: inputting the second feature into the PRNU model to obtain the fourth feature.

[0093] In some embodiments, the sensor pattern analysis method PRNU can be introduced after the ShuffleNet model described above to perform photoresponse non-uniformity analysis on the second feature. It should be noted that this photoresponse non-uniformity analysis can be used to detect differences in pixel sensitivity to light.

[0094] For example, the PRNU model can perform image acquisition and scaling processes for the input sources and noise of each input optical sensor.

[0095] In some embodiments, the image acquisition and scaling process for the input source and noise of the i-th input optical sensor can be performed as shown in the following formula (2).

[0096] y2=f(x+η j )+C o (2)

[0097] In formula (2), x can be the model input data, and y2 can be the second feature.

[0098] In addition, the η j It can be used to represent noise scaling. And C o It can be used to represent the initial values ​​for image acquisition.

[0099] S206, determine the first discrimination result based on the third feature, and determine the second discrimination result based on the fourth feature.

[0100] In some embodiments, determining a first discrimination result based on the third feature includes: inputting the third feature into an SVR model to obtain the first discrimination result; wherein the SVR model includes a plurality of preset non-sensitive region widths, each non-sensitive region width corresponds to a first discrimination value, and the first discrimination result is determined based on the plurality of first discrimination values.

[0101] For example, in this SVR model, the width of the non-sensitive region can be set first. The third feature can be represented as a feature vector, and the feature vector that falls within the non-sensitive region is not included in the loss. This disclosure does not limit the method of setting multiple non-sensitive region widths; for example, the size of each non-sensitive region width can be determined based on experience or application scenario.

[0102] For example, each SVR classification function can be obtained based on each non-sensitive region, where each SVR classification function can correspond to a first discriminant value. Then, the loss value corresponding to each SVR classification function can be determined, and the function weight corresponding to each first discriminant value can be determined by minimizing the loss value. It should be noted that each function weight needs to satisfy the following formula (3).

[0103]

[0104] In formula (3), n can be used to represent the number of non-sensitive areas. ∈i It can be used to represent the function weight corresponding to the i-th non-sensitive region. ∈i can be used to represent the width of the non-sensitive region corresponding to the i-th non-sensitive region.

[0105] Then, the first discrimination result can be calculated using the optimal value formula shown in formula (4).

[0106]

[0107] Among them, the W' SVR This can be the first discrimination result. 'n' can be used to represent the number of non-sensitive regions. 'w' ∈i It can be used to represent the function weight corresponding to the i-th non-sensitive region. i (x) can be used to represent the first discriminant value corresponding to the i-th non-sensitive region.

[0108] In some embodiments, determining a second discrimination result based on the fourth feature includes: inputting the fourth feature into an SVR model to obtain the second discrimination result; wherein the SVR model includes a plurality of preset non-sensitive region widths, each non-sensitive region width corresponds to a second discrimination value, and the second discrimination result is determined based on the plurality of second discrimination values.

[0109] It should be noted that the values ​​and quantities of the non-sensitive region widths used to calculate each second discrimination value can be the same as or different from the values ​​and quantities of the non-sensitive region widths used to calculate the first discrimination value. This disclosure does not limit this aspect.

[0110] In some embodiments, the method for determining the second discrimination result based on the fourth feature can be the same as the method for determining the first discrimination result described above, and will not be repeated here.

[0111] S208, determine the video detection result based on the first discrimination result and the second discrimination result.

[0112] In some embodiments, determining a video detection result based on the first discrimination result and the second discrimination result includes: determining a first evaluation index corresponding to the first discrimination result and a second evaluation index corresponding to the second discrimination result; and fitting a fusion ratio using the first evaluation index and the second evaluation index to determine the video detection result.

[0113] In an exemplary embodiment, the first evaluation index and the second evaluation index can be determined based on the F1 score evaluation index. For example, the video detection result can be calculated using the following formula (5).

[0114]

[0115] In formula (5), F 11 F can be the primary evaluation indicator. 12 This can be used as the second evaluation metric. `n` represents the number of non-sensitive regions defined for calculating the third characteristic. `m` represents the number of non-sensitive regions defined for calculating the fourth characteristic. `w` ∈j It can be used to represent the function weight corresponding to the j-th insensitive region. ∈j can be used to represent the width of the insensitive region corresponding to the j-th insensitive region.

[0116] The method provided in this disclosure can perform feature extraction by jointly using a first target network model and a second target network model, which can improve the sufficiency of feature extraction and retain more feature information. Furthermore, this disclosure introduces temporal consistency analysis and optical response non-uniformity analysis, thereby enabling the detection of various deepfake videos, improving the accuracy and reliability of deepfake video detection, and thus enhancing network security.

[0117] In an exemplary embodiment, a schematic diagram of the hybrid deep learning framework constructed in this disclosure embodiment can be as follows: Figure 3 As shown.

[0118] In Figure 3First, we can construct an InceptionResNet model and an Efficient-bo model. The InceptionResNet model uses the swish activation function, and the Efficient-bo model uses the ReLU rectified linear unit activation function. Then, we can concatenate and fuse the InceptionResNet and Efficient-bo models, and connect them to a BiLSTM model to perform temporal consistency analysis on the first feature to obtain the third feature. Finally, we can input the third feature into the SVR model to obtain the first discrimination result.

[0119] Furthermore, a ShuffleNet model can be constructed, and the second feature output by the ShuffleNet model can be analyzed using PRNU. Then, the fourth feature can be input into the SVR model to obtain the second discrimination result. Finally, the fusion ratio can be fitted using the F1 score evaluation metric, and the predicted value corresponding to this hybrid deep learning framework can be obtained; this predicted value is the video detection result.

[0120] In some embodiments, the present disclosure can also train the hybrid deep learning framework. In this case, the method further includes: acquiring a source video, preprocessing the source video to obtain frame images, and adding label information to each frame image to indicate that the corresponding image is a real image. Exemplarily, the step of preprocessing the source video may include face detection, image resizing, and image structure reshaping, etc.

[0121] Then, a deepfake image can be generated, and a label can be added to this deepfake image to indicate that the corresponding image is a deepfake. This deepfake image is then added to the aforementioned frame images. The aforementioned frame images are then split into training and testing sets with 75% used for training and 25% for testing. Sufficient training is then performed using cross-validation with a cross-validation value of 4.

[0122] For example, the cross-validation method can be k-fold cross-validation. Using cross-validation allows for repeated use of randomly generated subsamples for training and validation, effectively avoiding overlearning and underlearning states.

[0123] Then, the training set and test set can be input into a hybrid deep learning framework.

[0124] In some embodiments, the SVR model in a hybrid deep learning framework can be based on feature values ​​x, label information y, and a mapping function. The constructed result is the aforementioned tag information.

[0125] The SVR model can be represented by the following formula (6). Furthermore, the loss function corresponding to the SVR model can be represented by the following formula (7).

[0126]

[0127]

[0128] Here, w can be the normal vector. b can be the displacement term. y can take the value 1 or -1. When y is 1, it indicates that the image is a real image; when y is -1, it indicates that the image is a depth-spoofed image. Additionally, w determines the direction of the hyperplane, and b determines the distance between the hyperplane and the origin.

[0129] For example, a weighted decision is made by combining the first and second discrimination results output by the aforementioned hybrid deep learning framework to obtain the video detection result. At this point, manual inspection can also be performed to identify images with incorrect discrimination. These incorrectly discriminated images can be compiled into an error dataset, which can then be used for the next training iteration. Therefore, the training and test sets used for the next training iteration can be obtained by splitting the images from the source video (each frame), the generated deepfake images (each frame), and the images from the error dataset.

[0130] For example, the flowchart of a deepfake video detection method provided in this disclosure embodiment can be as follows: Figure 4 As shown.

[0131] In Figure 4 In this process, the input source video can first undergo preprocessing such as face detection, image resizing, and image structure reshaping. Then, deepfake images generated by the FOMM (First Order Motion Model) model are obtained, along with an error dataset. Finally, the preprocessed frames, deepfake images, and error dataset can be input into the hybrid deep learning framework.

[0132] This hybrid deep learning framework can consist of an IENet model, a ShuffleNet model, a BiLSTM model, PRNU analysis, and an SVR model. The IENet model serves as the first target network model. Then, the first and second discrimination results output by the hybrid deep learning framework are obtained, and a weighted decision is made to obtain the video detection result. Additionally, manual inspection can be performed based on the video detection results.

[0133] It should be noted that the embodiments of this disclosure can fully extract features and retain more feature information by parallelly fusing the first target network model and the second target network model and their corresponding activation functions. Furthermore, this disclosure serially fuses the two CNN models and the SVR model included in the first target network model, and simultaneously introduces light response non-uniformity analysis to construct a hybrid deep learning discriminative framework that weights and integrates the outputs of multiple models. This enables the identification of forged content generated using different face swapping and face replay techniques, improving the performance of the model's discrimination and enhancing the accuracy of deep fake video detection.

[0134] Based on the same inventive concept, this disclosure also provides a deepfake video detection device, as described in the following embodiments. Since the principle by which this device solves the problem is similar to that of the method embodiments described above, the implementation of this device embodiment can refer to the implementation of the method embodiments described above, and repeated details will not be repeated.

[0135] Figure 5 This diagram illustrates a deepfake video detection device according to an embodiment of the present disclosure, such as... Figure 5 As shown, the device includes:

[0136] The feature extraction module 501 is used to input an image set consisting of each frame image corresponding to the video to be detected into a first target network model to obtain a first feature, and input the image set into a second target network model to obtain a second feature. The first target network model and the second target network model are used to perform feature extraction based on different algorithms.

[0137] The feature analysis module 502 is used to perform time consistency analysis based on the first feature to obtain a third feature, and to perform optical response non-uniformity analysis based on the second feature to obtain a fourth feature.

[0138] The discrimination result determination module 503 is used to determine a first discrimination result based on the third feature and a second discrimination result based on the fourth feature;

[0139] The video detection result determination module 504 is used to determine the video detection result based on the first discrimination result and the second discrimination result.

[0140] In some embodiments of this disclosure, the feature analysis module 502 is used to input the first feature into the BiLSTM model to obtain the third feature.

[0141] In some embodiments of this disclosure, the feature analysis module 502 is used to input the second feature into the PRNU model to obtain the fourth feature.

[0142] In some embodiments of this disclosure, the first target network model is obtained by fusing two CNN models with different activation functions.

[0143] In some embodiments of this disclosure, the first target network model is obtained by fusing the InceptionResNet model and the Efficient-bo model, wherein the activation function corresponding to the InceptionResNet model is swish and the activation function corresponding to the Efficient-bo model is ReLU.

[0144] In some embodiments of this disclosure, the discrimination result determination module 503 is used to input the third feature into the SVR model to obtain the first discrimination result; wherein, the SVR model includes a plurality of preset non-sensitive region widths, each non-sensitive region width corresponds to a first discrimination value, and the first discrimination result is determined based on the plurality of first discrimination values.

[0145] In some embodiments of this disclosure, the video detection result determination module 504 is used to determine the first evaluation index corresponding to the first discrimination result and the second evaluation index corresponding to the second discrimination result; and to determine the video detection result by fitting a fusion ratio using the first evaluation index and the second evaluation index.

[0146] In some embodiments of this disclosure, the deepfake video detection apparatus provided in this disclosure further includes:

[0147] The image set determination module is used to acquire the video to be detected; and to perform frame extraction, face detection, and grayscale conversion on the video to be detected to obtain the image set.

[0148] The apparatus provided in this disclosure can perform feature extraction using a first target network model and a second target network model, thereby improving the sufficiency of feature extraction and retaining more feature information. Furthermore, this disclosure incorporates temporal consistency analysis and optical response non-uniformity analysis, enabling the detection of various deepfake videos, improving the accuracy and reliability of deepfake video detection, and thus enhancing network security.

[0149] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0150] The following reference Figure 6 To describe an electronic device 600 according to such an embodiment of the present disclosure. Figure 6 The electronic device 600 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0151] like Figure 6 As shown, the electronic device 600 is manifested in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, and a bus 630 connecting different system components (including storage unit 620 and processing unit 610).

[0152] The storage unit stores program code that can be executed by the processing unit 610, causing the processing unit 610 to perform the steps described in the "Detailed Description" section of this specification according to various exemplary embodiments of this disclosure.

[0153] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0154] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0155] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0156] Electronic device 600 can also communicate with one or more external devices 640 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. As shown, network adapter 660 communicates with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0157] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0158] In particular, according to embodiments of this disclosure, the process described above with reference to the flowchart can be implemented as a computer program product, which includes a computer program that, when executed by a processor, implements the above-described deepfake video detection method.

[0159] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. Figure 7 This illustration shows a schematic diagram of a computer-readable storage medium according to an embodiment of the present disclosure, such as... Figure 7 As shown, the computer-readable storage medium 700 stores a program product capable of implementing the methods described above. In some possible embodiments, various aspects of this disclosure may also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0160] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the methods described above is stored thereon. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code, which, when run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of this disclosure described in the "Detailed Description" section of this specification.

[0161] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0162] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0163] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0164] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0165] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0166] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0167] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0168] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope of this disclosure is indicated by the appended claims.

Claims

1. A method for detecting deepfake videos, characterized in that, include: The image set consisting of each frame of the video to be detected is input into the first target network model to obtain the first feature, and the image set is input into the second target network model to obtain the second feature. The first target network model and the second target network model are used to extract features based on different algorithms. A third feature is obtained by performing time consistency analysis based on the first feature, and a fourth feature is obtained by performing optical response non-uniformity analysis based on the second feature. The first discrimination result is determined based on the third feature, and the second discrimination result is determined based on the fourth feature; The video detection result is determined based on the first discrimination result and the second discrimination result.

2. The deepfake video detection method according to claim 1, characterized in that, The process of obtaining the third feature based on the first feature through time consistency analysis includes: The first feature is input into the Bidirectional Long Short-Term Memory (BiLSTM) network model to obtain the third feature.

3. The deepfake video detection method according to claim 1, characterized in that, The fourth feature is obtained by performing optical response non-uniformity analysis based on the second feature, including: The second feature is input into the PRNU model for optical response nonuniformity analysis to obtain the fourth feature.

4. The deepfake video detection method according to any one of claims 1 to 3, characterized in that, The first target network model is obtained by fusing two convolutional neural network (CNN) models with different activation functions.

5. The deepfake video detection method according to claim 4, characterized in that, The first target network model is obtained by fusing the InceptionResNet model and the Efficient-bo model, wherein the activation function corresponding to the InceptionResNet model is swish, and the activation function corresponding to the Efficient-bo model is the Modified Linear Unit (ReLU).

6. The deepfake video detection method according to any one of claims 1 to 3, characterized in that, Determining the first discrimination result based on the third feature includes: The third feature is input into the support vector regression (SVR) model to obtain the first discrimination result; The SVR model includes multiple preset non-sensitive region widths, each non-sensitive region width corresponds to a first discrimination value, and the first discrimination result is determined based on multiple first discrimination values.

7. The deepfake video detection method according to any one of claims 1 to 3, characterized in that, The step of determining the video detection result based on the first discrimination result and the second discrimination result includes: Determine the first evaluation index corresponding to the first discrimination result and the second evaluation index corresponding to the second discrimination result; The fusion ratio is obtained by fitting the first evaluation index and the second evaluation index to determine the video detection result.

8. The deepfake video detection method according to claim 1, characterized in that, Before inputting the image set consisting of each frame of the video to be detected into the first target network model to obtain the first feature, the method further includes: Acquire the video to be detected; The video to be detected is subjected to frame extraction, face detection, and grayscale conversion to obtain the image set.

9. A deepfake video detection device, characterized in that, include: The feature extraction module is used to input an image set consisting of each frame of the video to be detected into a first target network model to obtain a first feature, and input the image set into a second target network model to obtain a second feature. The first target network model and the second target network model are used to perform feature extraction based on different algorithms. The feature analysis module is used to perform time consistency analysis based on the first feature to obtain the third feature, and to perform optical response non-uniformity analysis based on the second feature to obtain the fourth feature. The discrimination result determination module is used to determine a first discrimination result based on the third feature and a second discrimination result based on the fourth feature; The video detection result determination module is used to determine the video detection result based on the first discrimination result and the second discrimination result.

10. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the deepfake video detection method according to any one of claims 1 to 8 by executing the executable instructions.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the deepfake video detection method according to any one of claims 1 to 8.

Images