A deep fake video detection method based on device fingerprint

Abstract

The present invention discloses a method for detecting deep fake videos based on device fingerprints, comprising the following steps: step 1, extracting fingerprint information of video shooting equipment; step 2, tracing the source of the video, by calculating the PCE value between the fingerprint information of the video to be detected and the fingerprint information of the reference video, so as to determine whether the same device is shooting; step 3, performing video authenticity detection by comparing the PCE value with a threshold. The method uses the device fingerprint information in the video to trace the source and detect the authenticity of the video, and uses the unchangeable device fingerprint information in the video, so that the proposed method has a certain robustness, provides a new idea for deep fake video detection, and has practical value.

Description

Technical Field

[0001] The present invention relates to the field of video detection technology, and in particular to a deep fake video detection method based on device fingerprint. Background Art

[0002] The rapid development of deep fake technology has provided convenience for art, visual effects and film and television entertainment, but it has also caused security risks such as financial fraud, cyber violence and political manipulation. Deep fake videos continue to spread on the Internet, threatening the security of cyberspace.

[0003] At present, in order to identify the authenticity of videos, researchers have proposed many effective methods. Existing detection methods are mainly divided into two categories: those based on deep neural networks and those based on manually extracted features. Among them, the deep learning detection method is mainly a well-designed feature extractor that can effectively capture the hidden patterns and features in deep fake videos; it adopts a complex deep learning network structure to better understand the spatiotemporal relationship in video sequences and improve the detection accuracy of fake videos. However, it is necessary to rely on a large amount of labeled sample data for training to enable the model to have better generalization ability. It is also necessary to introduce adversarial training to make the model more difficult to be disturbed by adversarial samples, thereby improving its performance in real scenes. In response to the needs of real-time applications, some methods are committed to improving the computational efficiency of deep learning models so that fast and real-time video authenticity detection can be performed in practical applications, but the methods are relatively complex and require the support of a large amount of data.

[0004] The second type of manual feature extraction method mainly draws on the idea of ​​image forensics. Although it can use some defects of forgery technology for detection, it faces challenges in accuracy and generalization when facing the ever-improving forgery technology. In the field of forged video detection, methods that rely on internal fingerprint features of images provide new ideas, but their accuracy is slightly insufficient and have strict requirements on data sets.

[0005] In summary, it can be seen that the existing forged video detection technology has two major defects and shortcomings:

[0006] (1) Video detection methods based on deep neural networks have a certain dependence on training data. Such methods are usually better when trained and tested in the same data set. However, in the absence of similar data for reference, the detection effect may drop sharply. Such methods may not be able to be effectively used when facing a single detection task. At the same time, deep learning detection methods have natural defects in interpretability.

[0007] (2) Existing deepfake related technologies are also constantly developing and improving. Therefore, although traditional manual feature extraction algorithms can use the defects of deepfake technology to collect evidence, as deepfake technology continues to fix its defects, such methods are no longer competent for related detection tasks, and the detection performance has dropped sharply. In addition, some manual feature extraction methods have strict requirements on data sets, and their generalization is relatively flawed. Summary of the invention

[0008] In view of the shortcomings of the existing technology, the present invention proposes a deep fake video detection method based on device fingerprint. The device fingerprint information in the video is used to trace the source and detect the authenticity of the video. The device fingerprint information in the video cannot be changed, so that the proposed method has a certain robustness, provides a new idea for deep fake video detection, and has practical value.

[0009] In order to solve the above technical problems, the technical solution of the present invention is:

[0010] A deep fake video detection method based on device fingerprint includes the following steps:

[0011] Step 1: Extract fingerprint information of video shooting device

[0012] Step 1.1: First, cut the captured video into frames, where the image expression of each frame is as follows:

[0013] I=I (0) +I (0) K+Θ

[0014] Among them, I (0) It is a noise-free pure image, K represents the fingerprint of the shooting device; Θ is a unified representative of all noises;

[0015] Step 1.2: Filter image I through a denoising filter to obtain a pure noise-free image.

[0016] Step 1.3: Noise-free image The noise residual R can be obtained by removing it from the original image i;

[0017] Step 1.4: Repeat steps 1.1 to 1.3 to obtain the noise residual R of several frames of images, and the noise residual R in the image can be regarded as the PRNU fingerprint of the shooting device. Combined with the M images taken by the shooting device, a more accurate PRNU fingerprint of the shooting device is estimated by the maximum likelihood method, that is, the shooting device fingerprint FP of the video, which is expressed as follows:

[0018]

[0019] Where R is the noise residual of the image and I is the image.

[0020] Step 2: Video tracing: Through step 1, the fingerprint information of the video shooting device is obtained from the video, and then the PCE value between the fingerprint information of the shooting device in the video to be detected and the fingerprint information of the shooting device in the reference video is calculated to determine whether they are shot by the same shooting device. The expression of the PCE value is as follows:

[0021]

[0022] The calculation steps in the formula are as follows:

[0023] 1) Calculate the fingerprint information of the camera at the peak position s peak The correlation ρ(s peak );

[0024] 2) Calculate the fingerprint information of the camera in the area The correlation ρ(s) at positions s other than ;

[0025] 3) All ρ(s) 2 Sum and use Perform normalization;

[0026] 4) The peak correlation ρ(s peak ) squared and divided by the normalized sum to obtain the final PCE value.

[0027] Where ρ(s peak ) represents the correlation of the fingerprint signal at the peak; ρ(s) represents the correlation of the fingerprint signal at position s; (m,n) is the dimension of the signal, such as the width and height of the image; is a set of positions that are excluded from the calculation, usually some specific areas, which means that the peak correlation PCE does not consider these positions.

[0028] Step 3: Video authenticity detection

[0029] Step 3.1: After determining the source of the video, the authenticity of the video can be tested by using the absence of fingerprint information in the facial area of ​​the video. The facial area of ​​the video to be tested and the reference video is cropped, and then the fingerprint information of the shooting device is extracted from the facial area through step 1;

[0030] Step 3.2, calculating the PCE value between the fingerprint information of the shooting device in the face area of ​​the video to be detected and the fingerprint information of the shooting device in the face area of ​​the reference video by the method of step 2;

[0031] Step 3.3, set the threshold. The threshold is set through experiments. There is a significant difference in the calculated PCE value between real videos and fake videos. If the PCE value exceeds the set threshold, the video to be detected is judged as a real video; conversely, if the PCE value does not reach the threshold, the video to be detected is judged as a deep fake video.

[0032] Preferably, the noise residual R includes PRNU noise and random noise.

[0033] Preferably, in step 2, firstly, a reference data set is constructed, wherein the reference data set includes source information of all videos in the known reference data set and source information of all unknown videos;

[0034] In the case of known source information, all frame images of the video to be detected are extracted and processed to obtain the fingerprint information of the shooting device of the video to be detected. At the same time, all videos in the reference data set are processed in the same way to obtain the fingerprint information of the shooting device of each reference video;

[0035] Subsequently, the PCE value between the fingerprint information of the shooting device of the video to be detected and the fingerprint information of the shooting device of each reference video is calculated. If the PCE value calculated between the fingerprint of the video to be detected and the fingerprint information of the shooting device of a reference video is the largest, then the video to be detected and this reference video are considered to be shot by the same shooting device.

[0036] Preferably, in step 2, in the absence of a known reference data source, first, all videos in the reference data set are subjected to comprehensive frame extraction and PRNU fingerprint processing;

[0037] Then, a blind clustering process was performed by using different clustering algorithms, and the blind clustering results were obtained;

[0038] In the present invention, three blind clustering algorithms are used, namely, a blind image clustering method based on the normalized cut criterion, a blind image clustering method based on correlation and consistency, and a blind image clustering method based on sparse reconstruction theory. Specifically, for example, if there is a batch of data, and it is not clear how many categories the batch of data can be divided into, the algorithm is used to divide the batch of data into several categories based on the characteristic relationship between the batch of data, such as the similarity between fingerprint information. Since it is not clear how many categories the batch of data is at the beginning, and the final division result is completely obtained by the clustering algorithm, this process is called blind clustering.

[0039] Based on the results of blind clustering, the fingerprint information of the shooting device of the video to be detected is calculated with the fingerprint information of each category of shooting device in the results of blind clustering, and the category with the largest PCE value of the shooting device fingerprint information of the video to be detected is found, and the device that shot the video to be detected is regarded as the same category as the shooting device with the largest PCE value in the results of blind clustering.

[0040] Preferably, in step 2, in the absence of known source information, the blind clustering operation obtains a blind clustering result which is a reference video of known source information.

[0041] Preferably, in step 2, the reference video and the video to be detected are of the category with the largest fingerprint PCE value, and the video to be detected is regarded as being of the same category as the device shot by the reference video.

[0042] The present invention has the following characteristics and beneficial effects:

[0043] Using the above technical solution, the present invention proposes a new detection method that focuses on the absence of fingerprint information inside the video and uses this unique perspective to determine the authenticity of the video. This method not only helps to alleviate the data dependence and complexity problems that may exist in deep learning methods, but also provides a novel and effective way to identify the authenticity of videos. Our research aims to explore diversified video tracing and detection methods to provide more options and solutions for protecting the authenticity of audiovisual content.

[0044] There are two different tracing scenarios: one is with prior information, and the other is analysis without prior information. This division makes the method proposed in this invention more widely applicable and universal. This comprehensive and flexible method is designed to meet the needs of video tracing in different scenarios, so that the research can show excellent universality in the face of diverse scenarios.

[0045] In addition, the present invention adopts a blind recognition method, which is unique in that it uses the unchangeable features in the video to trace the source of the device. This method has a wide range of applicability. Whether it is a real video or a fake video, it can trace the source of the device by analyzing the specific features inherent in the video that are difficult to tamper with. This comprehensive and efficient blind recognition strategy enables the method of the present invention to perform well in dealing with a variety of video scenarios, and provides a reliable means for video authenticity identification. By fully exploring and utilizing the properties of the video itself, the method of the present invention provides a unique and feasible approach to deep fake video detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 This is a flow chart of an embodiment of a method for detecting deep fake videos based on device fingerprints of the present invention. DETAILED DESCRIPTION

[0048] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0049] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0050] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0051] Example 1

[0052] The present invention provides a deep fake video detection method based on device fingerprint, such as Figure 1 As shown, the following steps are included:

[0053] Step 1: Extract fingerprint information of video shooting device

[0054] Step 1.1: Extracting fingerprint information from video recording equipment. First, the obtained video needs to be cropped into frames. Each frame can be understood as consisting of three parts: noise-free image, fingerprint information, and noise interference. What can be obtained is the fingerprint information in the image.

[0055] Specifically, the composition of an image can be described by the following formula:

[0056] I=I (0) +I (0) K+Θ

[0057] Among them, I (0) It is a noise-free pure image, K represents the fingerprint of the shooting device, which contains the uniqueness of the camera sensor and is also the key to its being regarded as a camera fingerprint; Θ is a unified representative of all noises; and in order to obtain the camera fingerprint, we first need to obtain a noise-free pure image.

[0058] Step 1.2, f represents a well-designed denoising filter. By using this filter, a pure image can be obtained. Specifically, the noise-free image can be obtained by the following formula

[0059]

[0060] Step 1.3: Noise-free image The noise residual R can be obtained by removing it from the original image I, and the expression is as follows:

[0061]

[0062] Here, R includes PRNU noise and other random noise, so they are collectively referred to as noise residuals.

[0063] Step 1.4: Repeat steps 1.1 to 1.3 to obtain the noise residual R of several frames of images, and the noise residual R in the image can be regarded as the PRNU fingerprint of the shooting device. Combined with the M images taken by the shooting device, a more accurate PRNU fingerprint of the shooting device is estimated by the maximum likelihood method, that is, the fingerprint FP of the shooting device, which is expressed as follows:

[0064]

[0065] Where R is the noise residual of the image and I is the image.

[0066] Step 2: Video tracing

[0067] Specifically, a reference data set is first constructed, which includes two cases: the source information of all videos in the reference data set is known and the source information of all videos is unknown. For the case of known source information, all frame images of the video to be detected are extracted and processed to obtain its shooting device fingerprint information. At the same time, all videos in the reference data set are similarly processed to obtain the shooting device fingerprint information of each reference video. Subsequently, by calculating the PCE value between the shooting device fingerprint information of the video to be detected and the shooting device fingerprint information of the reference video, the reference video with the largest PCE value between the shooting device fingerprint of the video to be detected can be found. Such a matching process makes it possible to regard the video to be detected as being shot by the same shooting device as this reference video.

[0068] In this embodiment, there are fingerprint information of two video shooting devices: FP1 and FP2. The fingerprints here are generally represented by a two-dimensional matrix, which are calculated using the PEC formula to obtain a specific value. If the two videos are shot by the same device, the PCE value calculated by them can reach more than 10,000; if the two videos are not shot by the same shooting device, the calculated PCE value can only reach tens to hundreds. The final value is determined by different models of shooting devices, but it will not reach thousands or tens of thousands.

[0069] In the above technical solution, the PCE value between the fingerprint information of the shooting device in the video to be detected and the fingerprint information of the shooting device in the reference video is calculated to determine whether they are shot by the same shooting device. The expression of the PCE value is as follows:

[0070]

[0071] The calculation steps in the formula are as follows:

[0072] 1) Calculate the fingerprint information of the camera at the peak position s peak The correlation ρ(s peak );

[0073] 2) Calculate the fingerprint information of the camera in the area The correlation ρ(s) at positions s other than ;

[0074] 3) All ρ(s) 2 Sum and use Perform normalization;

[0075] 4) The peak correlation ρ(s peak ) squared and divided by the normalized sum to obtain the final PCE value.

[0076] Where ρ(s peak) represents the correlation of the fingerprint signal at the peak; ρ(s) represents the correlation of the fingerprint signal at position s; (m,n) is the dimension of the signal, such as the width and height of the image; is a set of positions that are excluded from the calculation, usually some specific areas, which means that the peak correlation PCE does not consider these positions.

[0077] Step 3: Video authenticity detection

[0078] Step 3.1: After determining the source of the video, the authenticity of the video can be tested by using the absence of fingerprint information in the facial area of ​​the video. The facial area of ​​the video to be tested and the reference video is cropped, and then the device fingerprint information is extracted from the facial area through step 1;

[0079] Step 3.2: Calculate the PCE value between the facial region fingerprint information of the video to be detected and the facial region fingerprint of the reference video by the method of step 2. The calculation of the PCE value helps to quantify the similarity between the two, and then evaluate the authenticity of the video.

[0080] Step 3.3, set the threshold. The setting of the threshold is generally obtained through experiments. There is a significant difference in the calculated PCE value between real videos and fake videos. In this embodiment, after experiments, the threshold is set to 70. If the calculated PCE value is greater than 70, the video to be detected is judged as a real video; conversely, if the PCE value is less than 70, it is judged as a deep fake video.

[0081] Example 2

[0082] The difference between this embodiment and embodiment 1 is that in step 2, when there is no known reference data source, first, all videos in the reference data set are subjected to comprehensive frame extraction and PRNU fingerprint processing;

[0083] Then, a blind clustering process was performed by using different clustering algorithms, and the blind clustering results were obtained;

[0084] In the present invention, three blind clustering algorithms are used, namely, a blind image clustering method based on normalized cut criterion, a blind image clustering method based on correlation and consistency, and a blind image clustering method based on sparse reconstruction theory. Specifically, for example, if a batch of data is available and it is unclear how many categories the batch of data can be divided into, an algorithm is used to divide the batch of data into several categories based on the characteristic relationship between the batch of data, such as the similarity between fingerprint information. When it is unclear how many categories the batch of data is at the beginning, the final classification result is completely obtained by the clustering algorithm, and therefore, this process is called blind clustering.

[0085] Based on the results of blind clustering, the fingerprint information of the shooting device of the video to be detected is calculated with the fingerprint information of each category of shooting device in the results of blind clustering, and the category with the largest PCE value of the shooting device fingerprint information of the video to be detected is found, and the device that shot the video to be detected is regarded as the same category as the shooting device with the largest PCE value in the results of blind clustering.

[0086] In the above technical solution, due to the lack of reference information, in the absence of all video source information in the reference data set, a batch of videos will be faced with the challenge of dividing them according to device type. First, all videos in the reference data set are subjected to comprehensive frame extraction and PRNU fingerprint processing. Then, a blind clustering process is performed by using different clustering algorithms, because there is no prior knowledge at this stage. This blind clustering process is a classification without prior knowledge of device features, aiming to extract potential device type information from the data. Finally, the result of blind clustering is obtained. Based on this result, the fingerprint of the video to be detected is calculated with each type of fingerprint in the clustering result, and the type with the largest PCE value of the fingerprint of the video to be detected is found, and the device that shoots the video to be detected is regarded as the same type as the type of shooting device with the largest PCE value in the blind clustering result. Through this innovative method, the video to be detected can be identified as being shot by the same device as this type of video, thereby improving the accuracy and robustness of video authenticity detection in the absence of source information.

[0087] Specifically, for example, here we have the fingerprint information of the shooting devices of four videos: FP, FP1, FP2 and FP3, where FP is the device fingerprint information of the video to be detected, and the other three fingerprint information are the device fingerprint information of three different reference videos. The similarity between the three reference video fingerprint information is calculated to obtain the similarity matrix, as shown below:

[0088]

[0089] The first row represents FP1, the second row represents FP2, and the third row represents FP3. According to the blind clustering algorithm, these three fingerprint information can be divided into two categories: FP1 and FP2 are the first category, and FP3 is the second category. After that, the PCE value between FP and the two types of fingerprint information is calculated. If the PCE value calculated between FP and the first type of fingerprint information is larger, FP is considered to be the first category, otherwise FP is considered to be the second category. This can determine that the video to be detected is shot by the same device as the reference videos.

[0090] The other technical features of this embodiment are the same as those of Embodiment 1, and therefore are not described in detail in this embodiment.

[0091] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments including components are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A deep fake video detection method based on device fingerprint, characterized in that: The steps include: Step 1: Extract fingerprint information of video shooting device Step 1.1: First, cut the captured video into frames, where the image expression of each frame is as follows: I=I (0) +I (0) K+Θ Among them, I (0) It is a noise-free pure image, K represents the fingerprint of the shooting device; Θ is a unified representative of all noises; Step 1.2: Filter image I through a denoising filter to obtain a pure noise-free image. Step 1.3: Noise-free image The noise residual R can be obtained by removing it from the original image I; Step 1.4: Repeat steps 1.1 to 1.3 to obtain the noise residual R of several frames of images. The noise residual in the image is regarded as the PRNU fingerprint of the shooting device. Combined with the M images taken by the shooting device, the PRNU fingerprint of the shooting device is estimated by the maximum likelihood method. The PRNU fingerprint is the fingerprint information FP of the shooting device, and the expression is as follows: Where R is the noise residual of the image, I is the image; Step 2: Video tracing: Through step 1, the fingerprint information of the video shooting device is obtained from the video to be detected, and then the PCE value between the fingerprint information of the shooting device in the video to be detected and the fingerprint information of the shooting device in the reference video is calculated to determine whether they are shot by the same shooting device. The expression of the PCE value is as follows: The calculation steps in the formula are as follows: 1) Calculate the fingerprint information of the camera at the peak position s peak The correlation ρ(s peak ); 2) Calculate the correlation ρ(s) of the fingerprint information of the camera at the position s outside the area V; 3) All ρ(s) 2 Sum and use Perform normalization; 4) The peak correlation ρ(s peak ) squared and divided by the normalized sum to obtain the final PCE value; Among them, (m,n) is the dimension of the fingerprint information of the shooting device; is a set of locations that are excluded from the PCE value calculation; Step 3: Video authenticity detection Step 3.1, after determining the shooting source of the video to be detected, the authenticity of the video to be detected is detected by using the absence of fingerprint information in the facial area of ​​the video, that is, the facial area of ​​the video to be detected and the reference video is cropped, and then the fingerprint information of the shooting device is extracted from the facial area through step 1; Step 3.2, calculating the PCE value between the fingerprint information of the shooting device in the face area of ​​the video to be detected and the fingerprint information of the shooting device in the face area of ​​the reference video by the method of step 2; Step 3.3, set a threshold. If the PCE value calculated in step 3.2 exceeds the set threshold, the video to be detected is judged as a real video; conversely, if the PCE value calculated in step 3.2 does not reach the threshold, the video to be detected is judged as a deep fake video.

2. According to claim 1, a method for detecting deep fake videos based on device fingerprints is characterized in that: The noise residual R includes PRNU noise and random noise.

3. According to claim 2, a method for detecting deep fake videos based on device fingerprints is characterized in that: In step 2, firstly, a reference data set is constructed, wherein the reference data set includes all videos with known source information and all videos with unknown source information; When the source information of the reference video is known, all frame images of the video to be detected are extracted and processed to obtain the fingerprint information of the shooting device of the video to be detected. At the same time, all videos with known source information in the reference data set are processed in the same way to obtain the fingerprint information of the shooting device of each reference video. Subsequently, the PCE value between the fingerprint information of the shooting device of the video to be detected and the fingerprint information of the shooting device of each reference video is calculated. If the PCE value calculated between the fingerprint information of the shooting device of the video to be detected and the fingerprint information of the shooting device of a reference video is the largest, then the video to be detected and this reference video are considered to be shot by the same shooting device.

4. According to claim 3, a method for detecting deep fake videos based on device fingerprints is characterized in that: In step 2, when there is no video with known source information in the reference data set, first, all videos in the reference data set are comprehensively frame extracted and processed; Next, a blind clustering process is performed by using a variety of clustering algorithms to obtain the results of blind clustering; Based on the results of blind clustering, the fingerprint information of the shooting device of the video to be detected is calculated with the fingerprint information of each category of shooting device in the results of blind clustering, and the category with the largest PCE value of the shooting device fingerprint information of the video to be detected is found, and the device that shot the video to be detected is regarded as the same category as the shooting device with the largest PCE value in the results of blind clustering.

5. A method for detecting deep fake videos based on device fingerprint according to claim 4, characterized in that: The blind clustering process includes a normalized cut criterion, a clustering algorithm based on correlation and consistency, and a clustering algorithm based on sparse reconstruction.

Images