Deep video interpolation detection method based on inter-frame residual frequency reconstruction learning

Abstract

The present invention relates to the field of video detection technology, and discloses a deep video interpolation detection method based on inter-frame residual frequency reconstruction learning. Starting from the difference in inter-frame residual frequency features between the deep interpolation tampered video and the original video, a reconstruction strategy is used to learn the consistent representation of the residual frequency features of the original video. Specifically, reconstruction modules based on high-frequency and low-frequency frequency features of the inter-frame residual are designed respectively. The reconstruction module is forced to only fit the reconstruction process of the inter-frame residual frequency features of the original video to capture the consistent representation of the inter-frame residual of the original video. The difference between the intermediate features from different frequency reconstruction modules is used to construct a reconstruction difference guided attention module, so as to guide the backbone network to pay more attention to the difference between the original video and the deep interpolated video in the reconstruction process. The optimal performance is achieved in generalization tests such as cross-frame rate and cross-depth interpolation technology, and the detection effect is good.

Description

Technical Field

[0001] The present invention relates to the field of video detection technology, and in particular to a deep video interpolation detection method based on inter-frame residual frequency reconstruction learning. Background Art

[0002] Thanks to the rapid development of video generation technology, the visual quality of videos has significantly improved. Deep video interpolation is one such video enhancement technology. It uses deep learning to synthesize interpolated frames and insert them into the original frame sequence to optimize the motion continuity of the video. However, this technology can also be used maliciously. People may use deep interpolation to create fake high-frame-rate videos to gain more online clicks and illegal profits. It can also be used to repair abnormally deleted frames in surveillance evidence, thereby obscuring the truth. The malicious use of deep interpolation technology has had a negative impact on society, and more researchers are beginning to pay attention to deep video interpolation detection technology.

[0003] Xia et al. observed that most traditional interpolation methods introduce visual artifacts into the textured regions of the interpolated frames. Based on this observation, they proposed a two-stage interpolation detection method based on frame-level average texture change analysis. First, the average texture change value is calculated for each frame to obtain the average texture change curve of the video. The average texture change curve is then further processed to highlight its periodic characteristics, which can indicate the presence of interpolation in each frame and further estimate the original frame rate of the video. Ding et al. observed that there are differences in the motion residual signal between the original and interpolated videos. They proposed a passive forensic approach that transforms the identification of various traditional interpolation techniques into the problem of determining whether there are differences in their residual signals. First, a pre-classifier is designed to suppress the influence of the original and static interpolated frames in the video. Then, spatial and temporal Markov statistical features are extracted from the residual signal within the interpolated frames for the identification of traditional interpolation techniques. Experimental results show that the proposed model can effectively locate interpolated frames and further identify the traditional interpolation technique used. Ding et al. observed that deep learning-based interpolated videos have fewer visual tampering artifacts than traditional interpolated videos, making current traditional interpolation detection methods ineffective in detecting deep video interpolation tampering. Therefore, we propose a hybrid neural network that leverages prediction residuals and motion vector information in the coded domain for spatiotemporal representation learning to identify deep interpolated video. First, the residuals and motion vectors of moving regions are fixed using intra-frame prediction constraints. Then, the original residual and motion vector features are subjected to residual modulation and motion vector network refinement. The difference between the optimized features and the original features is used as the enhanced tampering trace. Finally, an attention-based two-stream network is designed to jointly learn discriminative representations from the enhanced traces.

[0004] Most existing video interpolation detection methods have achieved good results for traditional interpolation tampering and in-library testing. However, they perform poorly in the detection and generalization of deep interpolation tampering. First, due to inaccurate motion estimation, traditional interpolation technology will leave obvious visual artifacts in the tampered video. Such obvious tampering traces do not exist in the video generated by deep video interpolation technology. Therefore, traditional methods that focus too much on obvious visual artifacts cannot effectively detect deep interpolation tampering. Secondly, existing interpolation detection methods emphasize in-library detection effects. In-library detection refers to using a dataset with the same interpolation pattern for training and testing. They do not take into account the diverse situations of interpolation patterns. Therefore, these methods have reduced detection effects on videos with unknown interpolation patterns.

[0005] The prior art discloses a deep video interpolation detection method, comprising the following steps: extracting bidirectional optical flow and inter-frame motion anomaly regions from a video frame sequence; inputting the bidirectional optical flow into a ParNet network to extract local features; utilizing a motion attention mechanism to focus on inter-frame motion anomaly regions and enhance tampering characteristics; extracting single-frame motion anomaly regions from a video frame sequence and inputting them into a Transformer network for information reorganization, aggregating highly correlated information to highlight tampering traces and extracting global features; and aggregating the local and global features and inputting them into a fully connected layer for classification, outputting a classification result for true and false frames. This prior art emphasizes in-library detection and fails to consider the diverse nature of interpolation modes, resulting in poor detection results. Summary of the Invention

[0006] The purpose of the present invention is to provide a deep video interpolation detection method based on inter-frame residual frequency reconstruction learning, so as to solve the problem that the existing technology emphasizes the detection effect within the library, does not take into account the various situations of interpolation modes, and has poor detection effect.

[0007] To achieve the above objectives, the present invention provides a deep video interpolation detection method based on inter-frame residual frequency reconstruction learning, comprising:

[0008] S1. Obtain uncompressed videos to construct a dataset, compress the videos in the dataset, and divide the compressed videos into non-overlapping video segments. These non-overlapping video segments are the original video dataset. Perform deep interpolation on the original video dataset to obtain a deep interpolation tampered video dataset.

[0009] S2. Decode the videos in the original video dataset and the deep interpolation tampered video dataset into a series of RGB frames, scale the size of each frame and calculate the pixel difference between the previous and next frames as the inter-frame residual;

[0010] S3. Construct a backbone network, a reconstruction module, and a reconstruction difference-guided attention module, which together form a deep video interpolation detection model. The backbone network adopts Xception, and the reconstruction module adopts a U-shaped architecture. The reconstruction module includes a high-frequency reconstruction module and a low-frequency reconstruction module. Both the high-frequency reconstruction module and the low-frequency reconstruction module contain three downsampling modules and three upsampling modules. The features of the downsampling module and the upsampling module are spliced ​​along the channel dimension. The reconstruction difference-guided attention module is constructed by the intermediate difference of the intermediate features of the high-frequency reconstruction module and the low-frequency reconstruction module, as well as the upsampling layer and the downsampling layer.

[0011] S4. Input the inter-frame residuals into the reconstruction module, divide the input inter-frame residuals into high-frequency features and low-frequency features through discrete wavelet transform, and the high-frequency reconstruction module and low-frequency reconstruction module of the reconstruction module respectively model the high-frequency features and low-frequency features, and obtain the reconstructed inter-frame residuals through inverse discrete wavelet transform, and perform difference calculation on the inter-frame residuals and the reconstructed inter-frame residuals to obtain enhanced features;

[0012] S5. Input the enhanced features into the backbone network to obtain the main domain features, and input the main domain features, high-frequency features, and low-frequency features into the reconstruction difference-guided attention module to obtain the difference attention output features;

[0013] S6. Set the total loss function as the objective function. The total loss function includes reconstruction loss, metric learning loss, and cross-entropy classification loss. Adjust the weight parameters of the deep video interpolation detection model according to the objective function. Use the F1 score as the evaluation standard, test the obtained weight parameters, and obtain the final result, that is, the deep video interpolation detection result.

[0014] Preferably, in step S1, 36 uncompressed videos are obtained from an open source website to construct a data set, and the videos in the data set are compressed using the H.265 video encoder, with the bit rate and frame rate set to 2 Mb / s and 15 fps respectively, and each video is divided into non-overlapping video segments containing only 100 frames.

[0015] Preferably, in step S1, the depth interpolation methods include SepConv, EDSC, RRIN, IFRNet, and RIFE, and the target frame rates include 20fps, 25fps, 30fps, 60fps, and 120fps. Except for the target frame rate, the encoding parameters of all depth interpolation videos are consistent with the original video.

[0016] Preferably, in step S4, the formula for dividing the input inter-frame residual into high-frequency features and low-frequency features by discrete wavelet transform is as follows:

[0017]

[0018] in, is a function that performs mapping using the method indicated by the subscript, is the original inter-frame residual, and are the low-frequency and high-frequency features of the inter-frame residual, respectively.

[0019] Preferably, in step S4, the formula for modeling the high-frequency features and the low-frequency features to obtain the reconstructed low-frequency features and high-frequency features is:

[0020]

[0021]

[0022] in and are the reconstructed low-frequency features and high-frequency features respectively.

[0023] Preferably, the reconstructed inter-frame residual is obtained by inverse discrete wavelet transform The formula is as follows:

[0024]

[0025] Then the difference between the original inter-frame residual and the reconstructed inter-frame residual is calculated to obtain the feature The enhanced feature formula of the subsequent network is as follows:

[0026] .

[0027] Preferably, in step S5, the enhanced features are input into the backbone network to obtain the main domain features, and the main domain features, high frequency features and low frequency features are input into the reconstruction difference guided attention module. The specific process of obtaining the difference attention output features is to express the main features from the backbone network and the intermediate features from the low frequency and high frequency reconstruction modules as , and , the input features are transformed into , and Converted to five components, where the components Indicates the primary domain information. represents the query component, and Represents the key components from different frequency reconstruction modules. The correlation matrix between high-frequency and low-frequency query components and key components is obtained by the following calculation formula and ,

[0028] and

[0029] in, is the dot product;

[0030] Attention Map The difference between the high-frequency and low-frequency correlation matrices is normalized by the following formula:

[0031] ),

[0032] Then the attention map Acting on the main domain component The above outputs optimized features , the formula is as follows:

[0033] ,

[0034] Output features of the final module It is obtained by the following formula:

[0035] ,

[0036] in, are learnable weight parameters.

[0037] Preferably, in step S6, the reconstruction loss It consists of the absolute error between the high-frequency and low-frequency components of the original sample:

[0038]

[0039] in, The size of the batch input to the model, is the number of original videos in the batch;

[0040] The metric learning loss formula is as follows:

[0041]

[0042] in and are the intermediate feature pairs of the low-frequency and high-frequency reconstruction modules, respectively. These features are processed by global average pooling and L2 normalization. and are the collections of original videos and tampered videos respectively, and are the total number of (original, original) pairs and (original, tampered) pairs, respectively;

[0043] Cross entropy classification loss It is introduced as the objective function of the classification task, and the formula is as follows:

[0044]

[0045] in, is the label of the i-th video, is the predicted probability that the i-th video is considered to be a tampered video by the model;

[0046] Total loss function The formula is as follows:

[0047]

[0048] in is the weight parameter.

[0049] Preferably, in step S6, a majority voting mechanism is also included. For one video, the prediction results of the five video clips are voted, and the prediction result that accounts for the majority is used as the final result of the video.

[0050] Preferably, in step S2, the original video dataset and the deep interpolation tampered video dataset are divided into a training set and a test set for decoding, wherein the training set is a high frame rate of 30fps, 60fps and 120fps, and the test set is a low frame rate of 20fps and 25fps, and the training set and the test set use the same deep interpolation method;

[0051] In step S6, the best result when the frame rate of the training set and the test set and the depth interpolation method used are the same is selected as the final result of the in-library detection, the best result across frame rates is selected as the final result of the cross-frame rate detection, and the best result across frame rates and depth interpolation methods is selected as the final result of the cross-frame rate and depth interpolation method detection.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] First, starting from the difference in inter-frame residual frequency features between the deep interpolation tampered video and the original video, the reconstruction strategy is used to learn the consistent representation of the residual frequency features of the original video. Specifically, reconstruction modules based on the high-frequency and low-frequency frequency features of the inter-frame residual are designed respectively. The reconstruction module is forced to only fit the reconstruction process of the inter-frame residual frequency features of the original video to capture the consistent representation of the inter-frame residual of the original video. The difference between the intermediate features from different frequency reconstruction modules is used to construct a reconstruction difference guided attention module, which guides the backbone network to pay more attention to the difference between the original video and the deep interpolation video in the reconstruction process. The optimal performance is achieved in generalization tests such as cross-frame rate and cross-depth interpolation technology, and the detection effect is good. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 1 is a flow chart of a method for detecting deep video interpolation based on inter-frame residual frequency reconstruction learning according to an embodiment of the present invention;

[0055] Figure 2 Schematic diagram of RGB frames and inter-frame residuals of a deep video interpolation detection method based on inter-frame residual frequency reconstruction learning according to an embodiment of the present invention;

[0056] Figure 3 This is an overview of the model framework of the deep video interpolation detection method based on inter-frame residual frequency reconstruction learning according to an embodiment of the present invention;

[0057] Figure 4 1 is a diagram illustrating the reconstruction module architecture of a deep video interpolation detection method based on inter-frame residual frequency reconstruction learning according to an embodiment of the present invention;

[0058] Figure 5 This is a diagram of the architecture of a differential attention module of a deep video interpolation detection method based on inter-frame residual frequency reconstruction learning according to an embodiment of the present invention;

[0059] Figure 6 1 is a flow chart of the training process of a deep video interpolation detection method based on inter-frame residual frequency reconstruction learning according to an embodiment of the present invention;

[0060] Figure 7 This is a flow chart of the testing process of a deep video interpolation detection method based on inter-frame residual frequency reconstruction learning according to an embodiment of the present invention;

[0061] Figure 8 This is the robustness test result of the deep video interpolation detection method based on inter-frame residual frequency reconstruction learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0063] In the description of the present invention, it should be noted that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "back," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer" and the like, indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate the description of the present invention and simplify the description. They are not intended to indicate or imply that the devices or components referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0064] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0065] Furthermore, in the description of the present invention, unless otherwise specified, “plurality” means two or more.

[0066] Example 1

[0067] like Figure 1 、 2 As shown in FIG. 3 , a method for detecting deep video interpolation based on inter-frame residual frequency reconstruction learning in a preferred embodiment of the present invention includes:

[0068] S1. Obtain uncompressed videos to construct a dataset, compress the videos in the dataset, and divide the compressed videos into non-overlapping video segments. These non-overlapping video segments are the original video dataset. Perform deep interpolation on the original video dataset to obtain a deep interpolation tampered video dataset.

[0069] S2. Decode the videos in the original video dataset and the deep interpolation tampered video dataset into a series of RGB frames, scale the size of each frame and calculate the pixel difference between the previous and next frames as the inter-frame residual;

[0070] S3. Construct a backbone network, a reconstruction module, and a reconstruction difference-guided attention module, which together form a deep video interpolation detection model. The backbone network adopts Xception, and the reconstruction module adopts a U-shaped architecture. The reconstruction module includes a high-frequency reconstruction module and a low-frequency reconstruction module. Both the high-frequency reconstruction module and the low-frequency reconstruction module contain three downsampling modules and three upsampling modules. The features of the downsampling module and the upsampling module are spliced ​​along the channel dimension. The reconstruction difference-guided attention module is constructed by the intermediate difference of the intermediate features of the high-frequency reconstruction module and the low-frequency reconstruction module, as well as the upsampling layer and the downsampling layer.

[0071] S4. Input the inter-frame residuals into the reconstruction module, divide the input inter-frame residuals into high-frequency features and low-frequency features through discrete wavelet transform, and the high-frequency reconstruction module and low-frequency reconstruction module of the reconstruction module respectively model the high-frequency features and low-frequency features, and obtain the reconstructed inter-frame residuals through inverse discrete wavelet transform, and perform difference calculation on the inter-frame residuals and the reconstructed inter-frame residuals to obtain enhanced features;

[0072] S5. Input the enhanced features into the backbone network to obtain the main domain features, and input the main domain features, high-frequency features, and low-frequency features into the reconstruction difference-guided attention module to obtain the difference attention output features;

[0073] S6. Set the total loss function as the objective function. The total loss function includes reconstruction loss, metric learning loss, and cross-entropy classification loss. Adjust the weight parameters of the deep video interpolation detection model according to the objective function. Use the F1 score as the evaluation standard, test the obtained weight parameters, and obtain the final result, that is, the deep video interpolation detection result.

[0074] like Figure 4 As shown in the figure, a reconstruction module (RM) is introduced to model only the frequency features of the inter-frame residuals of the original video. This is mainly because there are many deep interpolation tampering methods. Mining the common features of the original video is more conducive to improving model generalization than overfitting the interpolation features of specific patterns in the training set. The output of the reconstruction module is then restored to the original dimension using the inverse DWT (IDWT). The reconstruction difference will reveal the tampering of the input features. Finally, the difference between the intermediate features of the high-frequency and low-frequency reconstruction modules is used to establish the differential attention (RDGA), further guiding the network to focus on the discriminative information in the reconstruction process, thereby achieving better detection performance.

[0075] The reconstruction module is as follows Figure 4 As shown in the figure, the reconstruction module is constructed using a U-shaped architecture. During the reconstruction process, the size of the convolution kernel in the time dimension is set to 3, and the stride and padding are both 1 to ensure that the time dimension of the intermediate features remains unchanged and that features from different frames can interact with each other. The downsampling layer reduces the spatial dimension of the feature map by half, while the upsampling layer uses transposed convolution to enlarge the spatial size of the feature map to twice the original spatial size. A three-dimensional convolution block is used in the reconstruction module because three-dimensional convolution is more effective in modeling the spatial features and temporal consistency of inter-frame residuals. The low-frequency reconstruction module has the same structure as the high-frequency reconstruction module, but its network parameters are respectively targeted at low-frequency features and high-frequency features.

[0076] This embodiment introduces a reconstruction strategy that forces the model to learn only the frequency reconstruction process of the original video, thereby enhancing the detection of tampering artifacts in videos with unknown interpolation patterns. Motion artifacts are typically visual blur and generally exist in the low-frequency domain. However, due to the unique design architecture of the deep interpolation model (using an upsampling module), it can inject abnormal high-frequency noise into the generated video frames. The reconstruction module learns the reconstruction process of different frequency features of the original video separately, allowing the model to fit a consistent representation of the original video features. It does not focus on the reconstruction learning process of the tampered video. Excessive focus on the characteristics of the tampered video can lead to overfitting to specific interpolation artifacts, preventing the model from improving the generalization of detection for unknown interpolation tampering patterns. A metric loss and a differential attention module are also designed to constrain the intermediate reconstruction features. The metric loss ensures semantic consistency between the high-frequency and low-frequency features of the original video during the reconstruction process. This can enhance the degree of distortion in the reconstructed tampered video, thereby highlighting the tampering artifacts. On the other hand, the differential attention module can use this constraint to make the model pay attention to whether the semantics of high-frequency features and low-frequency features in the reconstruction process are consistent, and supplement the backbone network with information on the semantic consistency of high-frequency and low-frequency features of the original video and the semantic inconsistency of high-frequency and low-frequency features of the tampered video, thereby improving the detection performance of the model.

[0077] Example 2

[0078] In step S1, 36 uncompressed videos are obtained from an open source website to construct a dataset. The videos in the dataset are compressed using the H.265 video encoder, with the bit rate and frame rate set to 2 Mb / s and 15 fps, respectively. Each video is split into non-overlapping video segments containing only 100 frames.

[0079] In this embodiment, 36 uncompressed videos from an open source website are used to construct a dataset.

[0080] In step S1, the depth interpolation methods include SepConv, EDSC, RRIN, IFRNet, and RIFE, and the target frame rates include 20fps, 25fps, 30fps, 60fps, and 120fps. Except for the target frame rate, the encoding parameters of all depth interpolation videos are consistent with the original video.

[0081] In this embodiment, the video is decoded into a series of RGB frames, and the size of each frame is scaled to 288×288 pixels.

[0082] In step S4, the formula for dividing the input inter-frame residual into high-frequency features and low-frequency features through discrete wavelet transform is as follows:

[0083]

[0084] in, is a function that performs mapping using the method indicated by the subscript, is the original inter-frame residual, and are the low-frequency and high-frequency features of the inter-frame residual, respectively.

[0085] In step S4, the formula for modeling the high-frequency features and the low-frequency features to obtain the reconstructed low-frequency features and high-frequency features is:

[0086]

[0087]

[0088] in and are the reconstructed low-frequency features and high-frequency features respectively.

[0089] Reconstructed inter-frame residuals are obtained by inverse discrete wavelet transform The formula is as follows:

[0090]

[0091] Then the difference between the original inter-frame residual and the reconstructed inter-frame residual is calculated to obtain the feature The enhanced feature formula of the subsequent network is as follows:

[0092] .

[0093] like Figure 5 As shown in Figure 3, the difference between the intermediate features of the high-frequency and low-frequency reconstruction modules is used to construct a differential attention module (RDGA), which aims to simulate the interaction between low-frequency tampering features and high-frequency tampering and strengthen the difference between the real video and tampered video features.

[0094] Example 3

[0095] In step S5, the enhanced features are input into the backbone network to obtain the main domain features, and the main domain features, high-frequency features and low-frequency features are input into the reconstruction difference-guided attention module. The specific process of obtaining the difference attention output features is to express the main features from the backbone network and the intermediate features from the low-frequency and high-frequency reconstruction modules as , and , the input features are transformed into , and Converted to five components, where the components Indicates the primary domain information. represents the query component, and Represents the key components from different frequency reconstruction modules. The correlation matrix between high-frequency and low-frequency query components and key components is obtained by the following calculation formula and ,

[0096] and

[0097] in, is the dot product;

[0098] Attention Map The difference between the high-frequency and low-frequency correlation matrices is normalized by the following formula:

[0099] ),

[0100] Then the attention map Acting on the main domain component The above outputs optimized features , the formula is as follows:

[0101] ,

[0102] Output features of the final module It is obtained by the following formula:

[0103] ,

[0104] in, are learnable weight parameters.

[0105] In step S6, the reconstruction loss It consists of the absolute error between the high-frequency and low-frequency components of the original sample:

[0106]

[0107] in, The size of the batch input to the model, is the number of original videos in the batch;

[0108] Reconstruction learning is a self-supervised learning method in which the encoder compresses high-dimensional input into key informative features during the reconstruction process. These features are crucial for distinguishing original and manipulated videos. To further enhance the difference between the reconstructed features of the original and manipulated videos, this paper introduces a metric learning loss. The metric learning loss guides the intermediate reconstructed features of the original and manipulated videos to approach each other during the reconstruction process, while the intermediate reconstructed features of the original and manipulated videos remain distant.

[0109] The metric learning loss formula is as follows:

[0110]

[0111] in and are the intermediate feature pairs of the low-frequency and high-frequency reconstruction modules, respectively. These features are processed by global average pooling and L2 normalization. and are the collections of original videos and tampered videos respectively, and are the total number of (original, original) pairs and (original, tampered) pairs, respectively;

[0112] Cross entropy classification loss It is introduced as the objective function of the classification task to guide the above reconstruction loss and measurement loss to have tampering forensics significance. The formula is as follows:

[0113]

[0114] in, is the label of the i-th video, is the predicted probability that the i-th video is considered to be a tampered video by the model;

[0115] Total loss function The formula is as follows:

[0116]

[0117] in is the weight parameter.

[0118] In the first 10 rounds of training, Set to 0.1 to promote the guiding role of classification loss, and after 10 rounds Set to 0.5.

[0119] Example 4

[0120] In step S6, in order to further improve the prediction stability of the model, a majority voting mechanism is also included. For a video, the prediction results of 5 video clips are voted, and the majority prediction result is used as the final result of the video. The specific training process and test process are as follows: Figure 6 and Figure 7 shown.

[0121] The detection task in this paper is a binary classification task, that is, determining whether a video is original or tampered. Here, the F1 score (the harmonic mean of precision and recall) is used.

[0122] As shown in Table 1, in step S2, the original video dataset and the deep interpolation tampered video dataset are divided into a training set and a test set for decoding. The training set uses high frame rates of 30fps, 60fps, and 120fps, and the test set uses low frame rates of 20fps and 25fps. The training set and the test set use the same deep interpolation method.

[0123] Table 1. Dataset division table

[0124]

[0125] In step S6, the best result when the frame rate of the training set and the test set and the depth interpolation method used are the same is selected as the final result of the in-library detection, the best result across frame rates is selected as the final result of the cross-frame rate detection, and the best result across frame rates and depth interpolation methods is selected as the final result of the cross-frame rate and depth interpolation method detection.

[0126] This example experiment compares the detection results of a baseline model (Xception) and two of the most representative methods in the field of interpolated frame detection (ST-MSF and LFDsN). Table 2 shows the detection results within the library. The interpolation method for the training and test sets is the same, and the frame rates are 30 fps, 60 fps, and 120 fps. The average metric is F1 (%). Bold font indicates the best result in this set of experimental results. Table 3 shows the detection results across frame rates. The interpolation method for the training and test sets is the same, but the training set frame rates are 30 fps, 60 fps, and 120 fps, while the test set frame rates are 20 fps and 25 fps. The average index is F1 (%), and bold font indicates that the result is the best in this set of experimental results; Table 4 shows the detection results across frame rates and interpolation methods. The interpolation methods of the training set and the test set are different, and the training set frame rates are 30fps, 60fps, and 120fps, and the test set frame rates are 20fps and 25fps. The test results are average values, and the average index is F1 (%). Bold font indicates that the result is the best in this set of experimental results. It can be seen that the detection performance of the present invention exceeds the existing technology in most cases; Table 5 is the ablation results of different modules proposed in the present invention, showing the contribution of different modules proposed in the present invention to the overall model. The interpolation methods and frame rates of the training set and the test set are the same. The average index is F1 (%). Bold font indicates that the result is the best in this set of experimental results. It can be seen that the module proposed in the present invention is effective in detecting deep interpolation tampering; Figure 8 The results of the robustness test are shown. For common video post-processing interference (video compression, Gaussian blur, saturation change, contrast transformation) scenarios, the present invention can still effectively identify deep frame insertion tampering videos.

[0127] Table 2. In-store testing results

[0128]

[0129] Table 3. Cross-frame rate detection results

[0130]

[0131] Table 4. Detection results of cross-frame rate and cross-frame interpolation methods

[0132]

[0133] Table 5. Ablation experiment results

[0134]

[0135] In summary, an embodiment of the present invention provides a deep video interpolation detection method based on inter-frame residual frequency reconstruction learning. Starting from the difference in inter-frame residual frequency features between the deep interpolation tampered video and the original video, the method uses a reconstruction strategy to learn the consistent representation of the residual frequency features of the original video. Specifically, reconstruction modules based on high-frequency and low-frequency frequency features of the inter-frame residual are designed respectively. The reconstruction module is forced to only fit the reconstruction process of the inter-frame residual frequency features of the original video to capture the consistent representation of the inter-frame residual of the original video. The difference between the intermediate features from different frequency reconstruction modules is used to construct a reconstruction difference guided attention module, so as to guide the backbone network to pay more attention to the difference between the original video and the deep interpolated video in the reconstruction process. The optimal performance is achieved in generalization tests such as cross-frame rate and cross-depth interpolation technology, and the detection effect is good.

[0136] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention. These improvements and substitutions should also be regarded as the scope of protection of the present invention.

Claims

1. A deep video interpolation detection method based on inter-frame residual frequency reconstruction learning, characterized in that: include: S1. Obtain uncompressed videos to construct a dataset, compress the videos in the dataset, and divide the compressed videos into non-overlapping video segments. These non-overlapping video segments are the original video dataset. Perform deep interpolation on the original video dataset to obtain a deep interpolation tampered video dataset. S2. Decode the videos in the original video dataset and the deep interpolation tampered video dataset into a series of RGB frames, scale the size of each frame and calculate the pixel difference between the previous and next frames as the inter-frame residual; S3. Construct a backbone network, a reconstruction module, and a reconstruction difference-guided attention module, which together form a deep video interpolation detection model. The backbone network adopts Xception, and the reconstruction module adopts a U-shaped architecture. The reconstruction module includes a high-frequency reconstruction module and a low-frequency reconstruction module. Both the high-frequency reconstruction module and the low-frequency reconstruction module contain three downsampling modules and three upsampling modules. The features of the downsampling module and the upsampling module are spliced ​​along the channel dimension. The reconstruction difference-guided attention module is constructed by the intermediate difference of the intermediate features of the high-frequency reconstruction module and the low-frequency reconstruction module, as well as the upsampling layer and the downsampling layer. S4. Input the inter-frame residuals into the reconstruction module, divide the input inter-frame residuals into high-frequency features and low-frequency features through discrete wavelet transform, and the high-frequency reconstruction module and low-frequency reconstruction module of the reconstruction module respectively model the high-frequency features and low-frequency features, and obtain the reconstructed inter-frame residuals through inverse discrete wavelet transform, and perform difference calculation on the inter-frame residuals and the reconstructed inter-frame residuals to obtain enhanced features; S5. Input the enhanced features into the backbone network to obtain the main domain features, and input the main domain features, high-frequency features, and low-frequency features into the reconstruction difference-guided attention module to obtain the difference attention output features; S6. Set the total loss function as the objective function. The total loss function includes reconstruction loss, metric learning loss, and cross-entropy classification loss. Adjust the weight parameters of the deep video interpolation detection model according to the objective function. Use the F1 score as the evaluation standard, test the obtained weight parameters, and obtain the final result, that is, the deep video interpolation detection result.

2. The method for deep video interpolation detection based on inter-frame residual frequency reconstruction learning according to claim 1, characterized in that: In step S1, 36 uncompressed videos are obtained from an open source website to construct a dataset. The videos in the dataset are compressed using the H.265 video encoder, with the bit rate and frame rate set to 2 Mb / s and 15 fps, respectively. Each video is split into non-overlapping video segments containing only 100 frames.

3. The method for deep video interpolation detection based on inter-frame residual frequency reconstruction learning according to claim 1, characterized in that: In step S1, the depth interpolation methods include SepConv, EDSC, RRIN, IFRNet, and RIFE, and the target frame rates include 20fps, 25fps, 30fps, 60fps, and 120fps. Except for the target frame rate, the encoding parameters of all depth interpolation videos are consistent with the original video.

4. The method for deep video interpolation detection based on inter-frame residual frequency reconstruction learning according to claim 1, characterized in that: In step S4, the formula for dividing the input inter-frame residual into high-frequency features and low-frequency features through discrete wavelet transform is as follows: in, is a function that performs mapping using the method indicated by the subscript, is the original inter-frame residual, and are the low-frequency and high-frequency features of the inter-frame residual, respectively.

5. The method for deep video interpolation detection based on inter-frame residual frequency reconstruction learning according to claim 1, characterized in that: In step S4, the formula for modeling the high-frequency features and the low-frequency features to obtain the reconstructed low-frequency features and high-frequency features is: in and are the reconstructed low-frequency features and high-frequency features respectively.

6. The method for deep video interpolation detection based on inter-frame residual frequency reconstruction learning according to claim 5, characterized in that: Reconstructed inter-frame residuals are obtained by inverse discrete wavelet transform The formula is as follows: Then the difference between the original inter-frame residual and the reconstructed inter-frame residual is calculated to obtain the feature The enhanced feature formula of the subsequent network is as follows: 。 7. The method for deep video interpolation detection based on inter-frame residual frequency reconstruction learning according to claim 1, characterized in that: In step S5, the enhanced features are input into the backbone network to obtain the main domain features, and the main domain features, high frequency features and low frequency features are input into the reconstruction difference guided attention module. The specific process of obtaining the difference attention output features is to express the main domain features from the backbone network and the intermediate features from the low frequency and high frequency reconstruction modules as , and , the input features are transformed into , and Converted to five components, where the components Indicates the primary domain information. represents the query component, and Represents the key components from different frequency reconstruction modules. The correlation matrix between high-frequency and low-frequency query components and key components is obtained by the following calculation formula and , and in, is the dot product; Attention Map The difference between the high-frequency and low-frequency correlation matrices is normalized by the following formula: ）， Then the attention map Acting on the main domain component The above outputs optimized features , the formula is as follows: ， Output features of the final module It is obtained by the following formula: ， in, are learnable weight parameters.

8. The method for deep video interpolation detection based on inter-frame residual frequency reconstruction learning according to claim 1, characterized in that: In step S6, the reconstruction loss It consists of the absolute error between the high-frequency and low-frequency components of the original sample: in, The size of the batch input to the model, is the number of original videos in the batch; The metric learning loss formula is as follows: in and are the intermediate feature pairs of the low-frequency and high-frequency reconstruction modules, respectively. These features are processed by global average pooling and L2 normalization. and are the collections of original videos and tampered videos respectively, and are the total number of original,original pairs and original,tampered pairs, respectively; Cross entropy classification loss It is introduced as the objective function of the classification task, and the formula is as follows: in, is the label of the i-th video, is the predicted probability that the i-th video is considered to be a tampered video by the model; Total loss function The formula is as follows: in is the weight parameter.

9. The method for deep video interpolation detection based on inter-frame residual frequency reconstruction learning according to claim 1, characterized in that: In step S6, a majority voting mechanism is also included. For a video, the prediction results of the five video clips are voted, and the prediction result that accounts for the majority is used as the final result of the video.

10. The method for deep video interpolation detection based on inter-frame residual frequency reconstruction learning according to claim 1, characterized in that: In step S2, the original video dataset and the deep interpolation tampered video dataset are divided into a training set and a test set for decoding. The training set uses high frame rates of 30fps, 60fps, and 120fps, and the test set uses low frame rates of 20fps and 25fps. The training set and the test set use the same deep interpolation method. In step S6, the best result when the frame rate of the training set and the test set and the depth interpolation method used are the same is selected as the final result of the in-library detection, the best result across frame rates is selected as the final result of the cross-frame rate detection, and the best result across frame rates and depth interpolation methods is selected as the final result of the cross-frame rate and depth interpolation method detection.

Images