Deepfake face-changing video detection method, system, storage medium and computing device based on neighborhood correlation time domain features

Abstract

The present invention discloses a Deepfake face-changing video detection method, system, storage medium and computing device based on neighborhood correlation time domain features. The method comprises the following steps: decoding the video to be tested into a frame sequence, determining the detection area of ​​the frame image; extracting the R, G, and B channel images of the detection area frame by frame to obtain the neighborhood correlation sequence of each frame image; constructing the neighborhood correlation matrix of the video to be tested; calculating the statistics of each column element of the neighborhood correlation matrix; calculating the neighborhood correlation difference matrix; calculating the statistics of each column element of the difference matrix; constructing a feature vector using each statistic; inputting the feature vector of the training set video into the classification model, classifying the video to be tested, and judging whether the video has been tampered with. The present invention utilizes the periodicity of the neighborhood correlation of the image after upsampling, and combines the time domain statistical features to capture the tampering traces of the Deepfake face-changing video, and has good robustness and real-time performance.

Description

Technical Field

[0001] The present invention relates to the technical field of digital video tampering detection, and in particular to a Deepfake face-changing video detection method, system, storage medium and computing device based on neighborhood correlation time-domain features. Background Art

[0002] With the rapid development of information technology, digital videos are widely used in news media, judicial evidence collection and other fields, so it is particularly important to ensure the authenticity of videos. However, the birth of Deepfake technology has made the threshold for tampering videos lower and lower. Users can synthesize realistic face-changing videos at a lower technical cost. The spread of these face-changing videos will pose a great threat to public opinion and personal identity information security. Therefore, the research on Deepfake face-changing video tampering detection technology is of great significance.

[0003] Existing detection methods are mainly divided into two categories according to the different feature extraction methods: one is to extract manual features based on traditional graphics methods, and the other is to extract features based on deep neural network methods. The typical method based on manual feature detection is to detect the authenticity of the video by extracting the spectral distribution of the video frame to be tested, or to use the feature inconsistency between the face area and the surrounding area to determine whether the video has been tampered with; the typical method based on deep neural network detection is to extract the spatiotemporal features of the video through cascading CNN (Convolutional Neural Networks) and LSTM (Long Short-Term Memory) for detection, or to use multi-branch network parallel connection to perform feature fusion to determine the authenticity of the video. Most of the methods based on manual features are based on single-frame image analysis, and the temporal features of the video are rarely considered; the detection accuracy of the method based on deep learning can usually be as high as 95% or more in the library, but the deep network model is more sensitive to input changes. When facing video compression and common image attacks (image scaling and image filtering, etc.), the algorithm detection performance is seriously reduced, and the algorithm robustness is poor. Summary of the invention

[0004] In order to overcome the defects and deficiencies of the prior art, the present invention provides a Deepfake face-changing video detection method based on neighborhood correlation time domain features. On the basis of fully considering the correlation between video frames, the present invention proposes to use the difference in the principle of video frame image generation to detect whether the video has been tampered with, and specifically uses the periodic characteristics of the neighborhood correlation of the image after the image is upsampled by the transposed convolution layer in the GAN (Generative Adversarial Networks) to determine whether the video has been tampered with. Generally speaking, the neighborhood correlation of the fake face image generated by GAN will be affected by the transposed convolution layer in the GAN network generation model. The upsampling operation performed by these transposed convolution layers will make the generated image present periodic neighborhood correlation, while the real video is generally directly recorded by a digital imaging device, so the neighborhood correlation of the real video frame image has no periodic regularity; on the other hand, when the real video is recorded and generated by a digital device, its adjacent frames have good continuity, so the neighborhood correlation of its adjacent frame images has good correlation, while the face-changing video is usually generated independently frame by frame by a specific face synthesis algorithm, and the neighborhood correlation of its adjacent frames may not have this regularity. Therefore, the present invention determines the authenticity of a video by calculating a variety of time-domain statistical features of a neighborhood correlation matrix of a video to be tested, and has good robustness and real-time performance.

[0005] The second purpose of the present invention is to provide a Deepfake face-changing video detection system based on neighborhood correlation time domain features.

[0006] A third object of the present invention is to provide a storage medium.

[0007] A fourth object of the present invention is to provide a computing device.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] A Deepfake face-changing video detection method based on neighborhood correlation time domain features comprises the following steps:

[0010] Decode the video to be tested into a frame sequence and determine the detection area of ​​the frame image;

[0011] Extract the R, G, and B channel images of the detection area frame by frame, calculate the neighborhood correlation feature parameters of each channel image, and connect them to obtain the neighborhood correlation sequence of each frame image;

[0012] Arrange the neighborhood correlation sequences of each frame row by row in chronological order to construct a neighborhood correlation matrix of the video to be tested;

[0013] Calculate the statistics of each column element of the neighborhood correlation matrix, wherein the statistics include standard deviation and kurtosis;

[0014] Performing difference operations on adjacent rows of the neighborhood correlation matrix to obtain a neighborhood correlation difference matrix;

[0015] Calculate the standard deviation statistics of each column element of the neighborhood correlation difference matrix;

[0016] Construct feature vectors using various statistics;

[0017] Input the feature vector of the training set video into the support vector machine training classification model, use the trained classification model to classify the video to be tested, and determine whether the video has been tampered with;

[0018] As a preferred technical solution, the steps of decoding the video to be tested into a frame sequence and determining the detection area of ​​the frame image include:

[0019] Use the FFmpeg tool to decode the video to be tested into a frame sequence with N frames. Then use the Dlib tool to detect the face area of ​​the image frame by frame, and reduce the rectangular frame position of the face area by 0.5 times toward the center to obtain the detection area.

[0020] As a preferred technical solution, the R, G, and B channel images of the detection area are extracted frame by frame, the neighborhood correlation feature parameters of each channel image are calculated, and the neighborhood correlation sequence of each frame image is obtained by connecting them. The specific steps include:

[0021] Extract the R, G, and B channel images of the detection area frame by frame and record them as and Where i represents the video frame number, and the Expectation-Maximization algorithm (EM) is used to extract and The corresponding D×D neighborhood correlation feature parameter (the neighborhood correlation does not need to consider the influence of the central pixel itself, so the central pixel does not participate in the calculation) is recorded as and Then and Expand it into a one-dimensional sequence in the order of row-by-row scanning, and connect it to obtain a 3×(D×D-1)-dimensional neighborhood correlation sequence k i .

[0022] The specific calculation process of EM is as follows:

[0023] For image I, if the current pixel can be obtained by convolution operation with its neighborhood pixels (that is, the reconstructed image I′ is close enough to the original image I), it is considered that the convolution kernel can characterize the neighborhood correlation of the image. At this time, the convolution kernel parameters of size D×D can be used as the neighborhood correlation feature parameters of image I.

[0024]

[0025] Where I'(x, y) represents the pixel value of the reconstructed image I' at (x, y), α = (D-1) / 2, k s,t is the value of the convolution kernel k at the neighborhood coordinate (s, t). At this time, the pixels of image I can be divided into two categories: (1) M1: image pixel value I (x, y) satisfies the above convolution formula; (2) M2: others. Here M1~N(0,σ 2 ), M2~U(a,b), where σ is the standard deviation of all pixels in image I, and a and b are the minimum and maximum values ​​of all pixels in image I respectively.

[0026] The solution of the convolution kernel k is obtained by alternating the expectation step and the maximum step. The expectation step is used to calculate the probability that each pixel in the image I belongs to the M1 class, and the maximum step is to update the convolution kernel parameter k by minimizing the error function E(k) s,t , the specific calculation formula is as follows:

[0027]

[0028] Where E(k) is the error function of the convolution kernel k, which is used to measure the difference between the reconstructed image I' and the original image I. s,t is the value of the convolution kernel k at the coordinate (s, t), w(x, y) is the weighting factor, indicating the probability that the current pixel belongs to the M1 class, and R(x, y) is the pixel value of the difference image R at the coordinate (x, y).

[0029] As a preferred technical solution, the neighborhood correlation sequences of each frame are arranged row by row in chronological order to construct a neighborhood correlation matrix of the video to be tested. The specific steps include:

[0030] Arrange the 3×(D×D-1)-dimensional neighborhood correlation sequences of each frame row by row in chronological order to obtain a neighborhood correlation matrix K (a two-dimensional matrix with N rows and 3×(D×D-1) columns) of the entire video to be tested.

[0031] As a preferred technical solution, the statistics of each column element of the neighborhood correlation matrix are calculated, and the statistics include standard deviation and kurtosis. The specific steps include:

[0032] For the neighborhood correlation matrix K, the statistics of each column of N elements are calculated, and the statistics include standard deviation and kurtosis. The calculation formulas of standard deviation statistics and kurtosis statistics are as follows:

[0033]

[0034] Where i represents the frame number, i∈[1,N], j represents the number of the feature dimension, j∈[1,3×(D×D-1)], and K(i,j) represents the element in the i-th row and j-th column of the neighborhood correlation matrix K. represents the mean of the eigenvalue sequence of the jth column of the convolution kernel parameter matrix K, S K (j) represents the standard deviation statistic S K The jth element of K (j) represents the standard deviation statistic P K The j-th element of .

[0035] As a preferred technical solution, the neighborhood correlation matrix is ​​subjected to a difference operation on adjacent rows to obtain a neighborhood correlation difference matrix, and the specific steps include:

[0036] For the neighborhood correlation matrix K, calculate its neighborhood correlation first-order difference matrix along the row direction of the matrix Neighborhood correlation second-order difference matrix and neighborhood correlation third-order difference matrix The calculation formula is as follows:

[0037]

[0038] Where K(n1:n2,:) represents the subset from the n1th row to the n2th row in the neighborhood correlation matrix K.

[0039] As a preferred technical solution, the specific steps of calculating the standard deviation statistics of each column element of the neighborhood correlation difference matrix include:

[0040] For the three neighborhood correlation difference matrices and calculate The standard deviation of each column of N-1 elements is calculated The standard deviation of each column of N-2 elements is calculated The standard deviation of each column of N-3 elements is obtained as a statistic and The specific calculation formula is as follows:

[0041]

[0042] As a preferred technical solution, the feature vector F is constructed by using various statistics.

[0043] In order to achieve the above second purpose, the present invention adopts the following technical solutions:

[0044] A Deepfake face-changing video detection system based on neighborhood correlation time-domain features, comprising: a video decoding module, a neighborhood correlation matrix calculation module, a neighborhood correlation matrix statistical feature calculation module, a neighborhood correlation difference matrix calculation module, a neighborhood correlation difference matrix statistical feature calculation module, a feature vector construction module, a classification model training module and a verification module;

[0045] The video decoding module is used to decode the video to be detected into a frame sequence;

[0046] The neighborhood correlation calculation module is used to determine the detection area of ​​each video frame image, then extract the R, G, and B channel images of the detection area, calculate the neighborhood correlation feature parameters of each channel image, and connect them to obtain the neighborhood correlation sequence of each frame image, and finally arrange the neighborhood correlation sequences of each frame row by row in chronological order to obtain the neighborhood correlation matrix of the video to be tested;

[0047] The neighborhood correlation matrix statistical feature calculation module is used to calculate the statistics of each column element of the neighborhood correlation matrix, and the statistics include standard deviation and kurtosis;

[0048] The neighborhood correlation difference matrix calculation module is used to perform a difference operation on the neighborhood correlation matrix along the row direction of the matrix to construct a neighborhood correlation difference matrix;

[0049] The neighborhood correlation difference matrix statistical feature calculation module is used to calculate the standard deviation statistics of each column element of the neighborhood correlation difference matrix;

[0050] The feature vector construction module is used to combine various statistics into feature values ​​to construct a feature vector;

[0051] The classification model training module is used to input the feature vector of the training sample into the support vector machine to train and generate the classification model;

[0052] The verification module is used to classify the video to be tested using the trained classification model to determine whether the video has been tampered with.

[0053] In order to achieve the third objective, the present invention adopts the following technical solutions:

[0054] A storage medium stores a program, which, when executed by a processor, implements the above-mentioned Deepfake face-changing video detection method based on neighborhood correlation time domain features.

[0055] In order to achieve the fourth objective, the present invention adopts the following technical solutions:

[0056] A computing device includes a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the above-mentioned Deepfake face-changing video detection method based on neighborhood correlation time domain features is implemented.

[0057] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0058] (1) The present invention fully considers the correlation between video frames and utilizes the differences in the principles of video frame image generation to detect whether the video has been tampered with. Specifically, it utilizes a variety of time-domain statistical features of the extracted neighborhood correlation matrix to determine whether the video has been tampered with, providing an effective way to detect tampering in face-changing videos.

[0059] (2) The present invention utilizes the inherent features in the image generation process for detection, which facilitates effective theoretical analysis and has good robustness and resistance to video compression.

[0060] (3) The algorithm used in the detection process of the present invention does not involve complex calculations and can be completed relatively quickly on a computer, with low cost, high efficiency and high practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 Schematic diagram of the process of Deepfake face-changing video detection method based on neighborhood correlation time-domain features of the present invention;

[0062] Figure 2 A schematic diagram of the location of the face area and the detection area of ​​the present invention;

[0063] Figure 3 Schematic diagram of a statistical histogram of the difference image R in the R color channel of the present invention;

[0064] Figure 4 Schematic diagram of a statistical histogram of the difference image R in the G color channel of the present invention;

[0065] Figure 5 Schematic diagram of a statistical histogram of the difference image R in the B color channel of the present invention;

[0066] Figure 6 Schematic diagram of the comparison of the neighborhood correlation matrix K of the real video and the face-swapped video of the present invention;

[0067] Figure 7 is the standard deviation statistic S of the real video and the face-swapped video of the present invention K A comparative diagram of

[0068] Figure 8 is the kurtosis statistic P of the real video and the face-swapped video of the present invention K A comparative diagram of

[0069] Fig. 9 is the first-order difference statistic of the real video and the face-changing video of the present invention A comparative diagram of

[0070] Fig.10 is the second-order difference statistic of the real video and the face-changing video of the present invention A comparative diagram of

[0071] Fig.11 is the third-order difference statistic of the real video and the face-changing video of the present invention A comparative diagram of

[0072] Fig.12 Schematic diagram of ROC curve of face-changing video tampering detection results at three compression rates of FaceForensics++ of the present invention. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0074] Example 1

[0075] like Figure 1 As shown, this embodiment provides a Deepfake face-changing video detection method based on neighborhood correlation time domain features. On the basis of fully considering the correlation between video frames, it is proposed to use the difference in the principle of video frame image generation to detect whether the video has been tampered with. Specifically, the periodic characteristics of the neighborhood correlation of the image after being upsampled by the transposed convolution layer in the GAN are used to determine whether the video has been tampered with. This embodiment takes the face tampering detection database FaceForensics++ as an example to introduce in detail the specific implementation process of the Deepfake face-changing video detection method based on neighborhood correlation time domain features, which specifically includes the following steps:

[0076] S1: Use FFmpeg to decode the video to be detected into a frame sequence of length N, and use Dlib to detect the face area, taking 0.5 times the face area as the detection area;

[0077] like Figure 2 As shown, first use the FFmpeg toolkit to decode the video and obtain all the video frame images F1, F2,,…, F N, the number of frames is N, and then the Dlib tool is used to detect the face area of ​​each frame image, and the rectangular frame position of the face area is reduced by 0.5 times to the center to obtain the detection area, where the reduction factor 0.5 is an empirical value, the purpose is to ensure that all pixels in the detection area are from the face-swapped image or the real image. The face area position is as follows Figure 2 As shown in the red dotted rectangle, the detection area is as follows Figure 2 As shown in the green solid rectangular box.

[0078] S2: Extract the R, G, and B channel images of the detection area frame by frame, calculate (including estimate) the neighborhood correlation feature parameters of each channel image, and connect them to obtain the neighborhood correlation sequence of each frame image. The specific steps are as follows:

[0079] Extract the R, G, and B channel images of the detection area frame by frame and record them as and Where i represents the video frame number, and the Expectation-Maximization algorithm (EM) is used to extract and The corresponding D×D neighborhood correlation feature parameter (the neighborhood correlation does not need to consider the influence of the central pixel itself, so the central pixel does not participate in the calculation) is recorded as and Then and Expand it into a one-dimensional sequence in the order of row-by-row scanning, and connect it to obtain a 3×(D×D-1)-dimensional neighborhood correlation sequence k i .

[0080] The specific calculation process of EM is as follows:

[0081] For image I, if the current pixel can be obtained by convolution operation with its neighborhood pixels (that is, the reconstructed image I′ is close enough to the original image I), it is considered that the convolution kernel can characterize the neighborhood correlation of the image. At this time, the convolution kernel parameters of size D×D can be used as the neighborhood correlation feature parameters of image I.

[0082]

[0083] Where I'(x, y) represents the pixel value of the reconstructed image I' at (x, y), α = (D-1) / 2, k s,t is the value of the convolution kernel k at the neighborhood coordinate (s, t). At this time, the pixels of image I can be divided into two categories: (1) M1: image pixel value I (x, y) satisfies the above convolution formula; (2) M2: others. Here M1~N(0,σ 2), M2~U(a,b), where σ is the standard deviation of all pixels in image I, and a and b are the minimum and maximum values ​​of all pixels in image I respectively.

[0084] like Figure 3 , Figure 4 and Figure 5 As shown, this embodiment gives the histogram distribution of the difference image corresponding to the 12th frame image of the face-changing video 019_018.mp4 in the FaceForensics++ database. It can be seen that the histograms of the difference image in the three channels of R, G, and B are mostly concentrated in the interval [0,3], indicating that the difference between the reconstructed image I' and the original image I is very small, that is, the convolution kernel k can better fit the neighborhood correlation of the original image I.

[0085] The solution of the convolution kernel k is obtained by alternating the expectation step and the maximum step. The expectation step is used to calculate the probability that each pixel in the image I belongs to the M1 class, and the maximum step is to update the convolution kernel parameter k by minimizing the error function E(k) s,t , the specific calculation formula is as follows:

[0086] E(k)=∑ x,y w(x,y)×R(x,y) 2 (2)

[0087]

[0088] Where E(k) is the error function of the convolution kernel k, which is used to measure the difference between the reconstructed image I' and the original image I. s,t is the value of the convolution kernel k at the coordinate (s, t), w(x, y) is the weighting factor, indicating the probability that the current pixel belongs to the M1 class, and R(x, y) is the pixel value of the difference image R at the coordinate (x, y).

[0089] S3: Arrange the neighborhood correlation sequences of each frame row by row in chronological order to construct a neighborhood correlation matrix of the video to be tested. The specific steps include:

[0090] Arrange the 3×(D×D-1)-dimensional neighborhood correlation sequences of each frame row by row in chronological order to obtain a neighborhood correlation matrix K (a two-dimensional matrix with N rows and 3×(D×D-1) columns) of the entire video to be tested.

[0091] like Figure 6As shown, this embodiment gives a schematic diagram comparing the neighborhood correlation matrices corresponding to the 1st frame to the 100th frame of the real video 019.mp4 and its corresponding face-swapped video 019_018.mp4 in the FaceForensics++ database. The results show that the neighborhood correlation matrix of the face-swapped video fluctuates more dramatically on the time axis.

[0092] S4: Calculate the statistics of each column element of the neighborhood correlation matrix, the statistics including standard deviation and kurtosis, and the specific steps include:

[0093] For the neighborhood correlation matrix K, the statistics of each column of N elements are calculated, and the statistics include standard deviation and kurtosis. The calculation formulas of standard deviation statistics and kurtosis statistics are as follows:

[0094]

[0095] Where i represents the frame number, i∈[1,N], j represents the number of the feature dimension, j∈[1,3×(D×D-1)], and K(i,j) represents the element in the i-th row and j-th column of the neighborhood correlation matrix K. represents the mean of the eigenvalue sequence of the jth column of the convolution kernel parameter matrix K, S K (j) represents the standard deviation statistic S K The jth element of K (j) represents the standard deviation statistic P K The j-th element of .

[0096] like Figure 7 , Figure 8 As shown, this embodiment gives the standard deviation statistics S of the real video 019.mp4 and its corresponding face-swapped video 019_018.mp4 in the FaceForensics++ database K and kurtosis statistic P K ,The results show that the standard deviation and kurtosis corresponding to the face-swapped ,video are higher than those of the real video, and there is a significant ,difference between the two.

[0097] S5: performing a difference operation on adjacent rows of the neighborhood correlation matrix to obtain a neighborhood correlation difference matrix. The specific steps include:

[0098] For the neighborhood correlation matrix K, calculate its neighborhood correlation first-order difference matrix along the row direction of the matrix Neighborhood correlation second-order difference matrix and neighborhood correlation third-order difference matrix The calculation formula is as follows:

[0099]

[0100] Where K(n1:n2,:) represents the subset from the n1th row to the n2th row in the neighborhood correlation matrix K.

[0101] S6: Calculate the standard deviation statistics of each column element of the neighborhood correlation difference matrix. The specific steps include:

[0102] For the three neighborhood correlation difference matrices and calculate The standard deviation of each column of N-1 elements is calculated The standard deviation of each column of N-2 elements is calculated The standard deviation of each column of N-3 elements is obtained as a statistic and The specific calculation formula is as follows:

[0103]

[0104] like Fig. 9 , Fig.10 and Fig.11 As shown, this embodiment gives the statistics of the real video 019.mp4 and its corresponding face-changing video 019_018.mp4 in the FaceForensics++ database. and The results show that the statistics corresponding to the face-swapped videos are higher than those of the real videos, and the difference between the two gradually increases with the increase of the difference order.

[0105] S7: constructing a feature vector using various statistics, wherein the combination method adopts a front-to-back connection method;

[0106] In order to improve the effectiveness of the algorithm, this embodiment combines several statistics in steps S4 and S6 to construct the feature vector F. The convolution kernel size D in this embodiment is 3. K , P K , and The dimension of the statistics is 3×(3×3-1)=24, so the dimension of the feature vector constructed in this embodiment is 24×5=120.

[0107] S8: Input the feature vectors of all training videos into a support vector machine to generate a classification model, and use the model to classify the test video to determine whether the video has been tampered with.

[0108] In this embodiment, the FaceForensics++ database (Deepfake part) contains 1000 pairs of c0 lossless real and fake face videos, 1000 c23 low-compression fake face videos, and 1000 c40 high-compression fake face videos. This embodiment randomly selects 500 pairs of c0 lossless real and fake face videos as training sets to train the model, selects the remaining 500 pairs of c0 lossless videos, c23 low-compression videos, and c40 high-compression videos as test sets, and uses SVM to train and test the data. This embodiment uses the libsvm toolkit for classification, and the classification process is as follows:

[0109] 1. First, obtain the feature vectors of all videos;

[0110] 2. Normalize the feature vector of the training set and save the scaling parameter file norm.scale

[0111] 3. Cross-validate the training set data and select the best penalty coefficient c and kernel function parameter γ;

[0112] 4. Use the optimal parameters c and γ to train the training set data to obtain the training model Model;

[0113] 5. Use the parameter file norm.scale to normalize the test feature vector, and then use the model Model to classify the test data to determine whether the video has been tampered with.

[0114] like Fig.12 As shown in FIG. 1 , the test results of the ROC (receiver operating characteristic) curves for the three compression rate videos, and the ACC (accuracy) and AUC (area under the ROC curve) are shown in Table 1. The results show that the detection effect of the present invention under the three compression rates has no obvious fluctuation, which proves that the present invention has strong robustness for the detection of tampered videos with different compression rates.

[0115] Table 1 FaceForensics++ database test results

[0116] Test Data Lossless Video c0 Low compression video c23 High compression video c40 ACC 0.860 0.851 0.852 AUC 0.918 0.908 0.911

[0117] On the basis of fully considering the correlation between video frames, the present invention proposes to use the difference in the principle of video frame image generation to detect whether the video has been tampered with, and specifically uses the characteristics of the periodic neighborhood correlation of the image caused by the upsampling operation of the transposed convolution layer in GAN (Generative Adversarial Networks) to determine whether the video has been tampered with. In general, the neighborhood correlation of the fake face image generated by GAN will be affected by the transposed convolution layer in the GAN network generation model. The upsampling operation performed by these transposed convolution layers will make the generated image present a periodic neighborhood correlation, while the real video is generally directly recorded by a digital imaging device, so the neighborhood correlation of the real video frame image has no periodic regularity. On the other hand, when the real video is recorded and generated by a digital device, its adjacent frames have good continuity, so the neighborhood correlation of its adjacent frame images has good correlation, while the face-changing video is usually generated independently frame by frame by a specific face synthesis algorithm, and the neighborhood correlation of its adjacent frames may not have this regularity. Therefore, the present invention determines the authenticity of the video by calculating a variety of time-domain statistical features of the neighborhood correlation matrix to be processed, and has good robustness and real-time performance.

[0118] Example 2

[0119] The present embodiment provides a Deepfake face-changing video detection system based on neighborhood correlation time-domain features, including: a video decoding module, a neighborhood correlation matrix calculation module, a neighborhood correlation matrix statistical feature calculation module, a neighborhood correlation difference matrix calculation module, a neighborhood correlation difference matrix statistical feature calculation module, a feature vector construction module, a classification model training module and a verification module, and the implementation steps of each module are the same as the method in Example 1;

[0120] In this embodiment, the video decoding module is used to decode the video to be detected into a frame sequence;

[0121] In this embodiment, the neighborhood correlation calculation module is used to determine the detection area of ​​each video frame image, then extract the R, G, and B channel images of the detection area, calculate (including estimate) the neighborhood correlation feature parameters of each channel image, and connect them to obtain the neighborhood correlation sequence of each frame image, and finally arrange the neighborhood correlation sequences of each frame row by row in chronological order to obtain the neighborhood correlation matrix of the video to be tested;

[0122] In this embodiment, the neighborhood correlation matrix statistical feature calculation module is used to calculate the statistics of each column element of the neighborhood correlation matrix, and the statistics include standard deviation and kurtosis;

[0123] The neighborhood correlation difference matrix calculation module is used to perform a difference operation on the neighborhood correlation matrix along the row direction of the matrix to construct a neighborhood correlation difference matrix;

[0124] The neighborhood correlation difference matrix statistical feature calculation module is used to calculate the standard deviation statistics of each column element of the neighborhood correlation difference matrix;

[0125] In this embodiment, the feature vector construction module is used to construct a feature vector using various statistics;

[0126] In this embodiment, the classification model training module is used to input the feature vector of the training sample into the support vector machine to train and generate the classification model;

[0127] In this embodiment, the verification module is used to classify the video to be tested using the trained classification model to determine whether the video has been tampered with.

[0128] Example 3

[0129] This embodiment provides a storage medium, which may be a storage medium such as ROM, RAM, disk, or CD. The storage medium stores one or more programs. When the program is executed by a processor, the Deepfake face-changing video detection method based on neighborhood correlation time domain features of Embodiment 1 is implemented.

[0130] Example 4

[0131] The present embodiment provides a computing device, which may be a desktop computer, a laptop computer, a smart phone, a PDA handheld terminal, a tablet computer or other terminal device with a display function. The computing device includes a processor and a memory, and the memory stores one or more programs. When the processor executes the program stored in the memory, the Deepfake face-changing video detection method based on neighborhood correlation time domain features of Example 1 is implemented.

[0132] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A Deepfake face-changing video detection method based on neighborhood correlation time domain features, characterized in that: The steps include: Decode the video to be tested into a frame sequence and determine the detection area of ​​the frame image; Extract the R, G, and B channel images of the detection area frame by frame, calculate the neighborhood correlation feature parameters of each channel image, and connect them to obtain the neighborhood correlation sequence of each frame image; Arrange the neighborhood correlation sequences of each frame row by row in chronological order to construct a neighborhood correlation matrix of the video to be tested; Calculate the statistics of each column element of the neighborhood correlation matrix, wherein the statistics include standard deviation and kurtosis; Performing difference operations on adjacent rows of the neighborhood correlation matrix to obtain a neighborhood correlation difference matrix; Calculate the standard deviation statistics of each column element of the neighborhood correlation difference matrix; Construct feature vectors using various statistics; The feature vectors of the training set videos are input into the support vector machine training classification model, and the trained classification model is used to classify the video to be tested to determine whether the video has been tampered with.

2. The Deepfake face-changing video detection method based on neighborhood correlation time domain features according to claim 1 is characterized in that: The specific steps of decoding the video to be tested into a frame sequence and determining the detection area of ​​the frame image include: Use the FFmpeg tool to decode the video to be tested into a frame sequence with N frames. Then use the Dlib tool to detect the face area of ​​the image frame by frame, and reduce the rectangular frame position of the face area by 0.5 times toward the center to obtain the detection area.

3. The Deepfake face-changing video detection method based on neighborhood correlation time domain features according to claim 1 is characterized in that: The R, G, and B channel images of the detection area are extracted frame by frame, the neighborhood correlation feature parameters of each channel image are calculated, and the neighborhood correlation feature parameters are connected to obtain the neighborhood correlation sequence of each frame image. The specific steps include: Extract the R, G, and B channel images of the detection area frame by frame and record them as and Where i represents the video frame number, and the maximum expectation algorithm EM is used to extract and The corresponding D×D neighborhood correlation feature parameter is recorded as and Then and Expand it into a one-dimensional sequence in the order of row-by-row scanning, and connect it to obtain a 3×(D×D-1)-dimensional neighborhood correlation sequence k i ; The specific calculation process of EM is as follows: For image I, if the current pixel can be obtained by convolution operation with its neighboring pixels, it is considered that the convolution kernel can characterize the neighborhood correlation of the image. At this time, the convolution kernel parameters of size D×D are used as the neighborhood correlation feature parameters of image I. Where I'(x, y) represents the pixel value of the reconstructed image I' at (x, y), α = (D-1) / 2, k s,t is the value of the convolution kernel k at the neighborhood coordinate (s, t). At this time, the pixels of image I can be divided into two categories: (1) M1: image pixel value I (x, y) satisfies the above convolution formula; (2) M2: others. Here M1~N(0,σ 2 ), M2~U(a,b), where σ is the standard deviation of all pixels in image I, a and b are the minimum and maximum values ​​of all pixels in image I respectively; The solution of the convolution kernel k is obtained by alternating the expectation step and the maximum step. The expectation step is used to calculate the probability that each pixel in the image I belongs to the M1 class, and the maximum step is to update the convolution kernel parameter k by minimizing the error function E(k). s,t , the specific calculation formula is as follows: Where E(k) is the error function of the convolution kernel k, which is used to measure the difference between the reconstructed image I' and the original image I. s,t is the value of the convolution kernel k at the coordinate (s, t), w(x, y) is the weighting factor, indicating the probability that the current pixel belongs to the M1 class, and R(x, y) is the pixel value of the difference image R at the coordinate (x, y).

4. The Deepfake face-changing video detection method based on neighborhood correlation time domain features according to claim 1 is characterized in that: The steps of arranging the neighborhood correlation sequences of each frame row by row in time order to construct a neighborhood correlation matrix of the video to be tested include: Arrange the 3×(D×D-1)-dimensional neighborhood correlation sequences of each frame row by row in chronological order to obtain the neighborhood correlation matrix K of the entire video to be tested. K is a two-dimensional matrix with N rows and 3×(D×D-1) columns.

5. The Deepfake face-changing video detection method based on neighborhood correlation time domain features according to claim 1 is characterized in that: The statistics of each column element of the neighborhood correlation matrix are calculated, wherein the statistics include standard deviation and kurtosis. The specific steps include: For the neighborhood correlation matrix K, the statistics of each column of N elements are calculated, including standard deviation and kurtosis. The standard deviation statistic S K and kurtosis statistic P K The calculation formula is as follows: Where i represents the frame number, i∈[1,N], j represents the number of the feature dimension, j∈[1,3×(D×D-1)], and K(i,j) represents the element in the i-th row and j-th column of the neighborhood correlation matrix K. represents the mean of the eigenvalue sequence of the jth column of the convolution kernel parameter matrix K, S K (j) represents the standard deviation statistic S K The jth element of K (j) represents the standard deviation statistic P K The j-th element of .

6. The Deepfake face-changing video detection method based on neighborhood correlation time domain features according to claim 1 is characterized in that: The neighborhood correlation matrix is ​​subjected to difference operation of adjacent rows to obtain a neighborhood correlation difference matrix. The specific steps include: For the neighborhood correlation matrix K, calculate its neighborhood correlation first-order difference matrix along the row direction of the matrix Neighborhood correlation second-order difference matrix and neighborhood correlation third-order difference matrix The calculation formula is as follows: Wherein K(n1:n2,:) represents the subset from the n1th row to the n2th row in the neighborhood correlation matrix K; the step of calculating the standard deviation statistic of each column element of the neighborhood correlation difference matrix comprises: For the three neighborhood correlation difference matrices and calculate The standard deviation of each column of N-1 elements is calculated The standard deviation of each column of N-2 elements is calculated The standard deviation of each column of N-3 elements is obtained as a statistic and The specific calculation formula is as follows:

7. The Deepfake face-changing video detection method based on neighborhood correlation time domain features according to claim 1 is characterized in that: The feature vector F is constructed by using various statistics.

8. A Deepfake face-changing video detection system based on neighborhood correlation time domain features based on the detection method according to any one of claims 1 to 7, characterized in that: include: Video decoding module, neighborhood correlation matrix calculation module, neighborhood correlation matrix statistical feature calculation module, neighborhood correlation difference matrix calculation module, neighborhood correlation difference matrix statistical feature calculation module, feature vector construction module, classification model training module and verification module; The video decoding module is used to decode the video to be detected into a frame sequence; The neighborhood correlation calculation module is used to determine the detection area of ​​each video frame image, then extract the R, G, and B channel images of the detection area, estimate the neighborhood correlation feature parameters of each channel image, and connect them to obtain the neighborhood correlation sequence of each frame image, and finally arrange the neighborhood correlation sequences of each frame row by row in chronological order to obtain the neighborhood correlation matrix of the video to be tested; The neighborhood correlation matrix statistical feature calculation module is used to calculate the statistics of each column element of the neighborhood correlation matrix, and the statistics include standard deviation and kurtosis; The neighborhood correlation difference matrix calculation module is used to perform a difference operation on the neighborhood correlation matrix along the row direction of the matrix to construct a neighborhood correlation difference matrix; The neighborhood correlation difference matrix statistical feature calculation module is used to calculate the standard deviation statistics of each column element of the neighborhood correlation difference matrix; The feature vector construction module is used to construct a feature vector using various statistics; The classification model training module is used to input the feature vector of the training sample into the support vector machine to train and generate the classification model; The verification module is used to classify the video to be tested using the trained classification model to determine whether the video has been tampered with.

9. A storage medium storing a program, characterized in that: When the program is executed by a processor, the Deepfake face-changing video detection method based on neighborhood correlation time domain features as described in any one of claims 1 to 7 is implemented.

10. A computing device comprising a processor and a memory for storing a program executable by the processor, characterized in that: When the processor executes the program stored in the memory, the Deepfake face-changing video detection method based on neighborhood correlation time domain features as described in any one of claims 1 to 7 is implemented.

Images