A video frame interpolation detection method based on unsupervised pre-training and multi-scale fusion

Abstract

The application provides a video frame insertion detection method based on unsupervised pre-training and multi-scale fusion, and belongs to the technical field of image recognition, and specifically comprises the following steps: in the unsupervised pre-training stage, a continuous video frame sequence without labels is acquired, multi-scale visual features are extracted, feature fusion is performed through a feature pyramid network, and a unified feature map is generated. Then, a hybrid loss function is constructed through a reconstruction loss and a contrast loss, model parameters are optimized, and the encoder weight is saved. In the supervised fine-tuning stage, the encoder weight is loaded and frozen, features of the video frame with a binary classification label are extracted, mapped into a binary classification probability distribution, and the classifier and unfrozen parameters are optimized based on a cross-entropy loss. Finally, the fine-tuned model is used for feature extraction and classification reasoning, and the result of whether it is frame insertion tampering or not is output. Through the scheme of the application, the detection efficiency and accuracy are improved.

Description

A video frame interpolation detection method based on unsupervised pre-training and multi-scale fusion Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a video frame interpolation detection method based on unsupervised pre-training and multi-scale fusion. Background Technology

[0002] Currently, contemporary video frame rate upconversion (FRUC) forensic detection methods mainly employ temporal artifact analysis through two different paradigms: (1) traditional methods utilizing manually designed spatiotemporal features, and (2) deep neural networks with hierarchical feature abstraction. Traditional methods reveal tampering artifacts through manually designed forensic markers. However, traditional FRUC forensic detection methods face significant limitations in practical applications: First, the design logic of these methods highly depends on prior knowledge, and most are built on supervised learning frameworks, exhibiting a strong dependence on large-scale labeled data. However, in real forensic scenarios, due to the diversity of FRUC algorithms, the complexity of video content, and the high cost of labeling, it is difficult to obtain sufficient and comprehensive labeled datasets for model training, resulting in a sharp decline in the performance of the method on unseen samples. Second, existing traditional detection methods generally suffer from weak generalization ability across datasets and algorithms. When the detection target is switched to a video with different data distributions, or a video generated by an FRUC algorithm that has not participated in training, the discriminative power of the features will be greatly reduced, making it difficult to effectively identify tampering traces.

[0003] It is evident that there is an urgent need for a video frame interpolation detection method based on unsupervised pre-training and multi-scale fusion, which has high detection efficiency and accuracy. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a video frame interpolation detection method based on unsupervised pre-training and multi-scale fusion, which at least partially solves the problems of poor detection efficiency and accuracy in the prior art.

[0005] This invention provides a video frame interpolation detection method based on unsupervised pre-training and multi-scale fusion, comprising:

[0006] Step 1: In the unsupervised pre-training phase, obtain unlabeled continuous video frame sequences as input;

[0007] Step 2: Input each frame in the video frame sequence into a shared feature extraction network to extract multi-scale visual features for each frame;

[0008] Step 3: Input the multi-scale visual features into the Feature Pyramid Network (FPN) for cross-scale feature fusion to obtain the fused feature map for each frame.

[0009] Step 4: The fused feature maps of multiple frames are stitched together along the channel dimension and input into the spatiotemporal fusion module to obtain a unified feature map that fuses spatial details and temporal dynamic information. The spatiotemporal fusion module includes a channel-space attention module (CBAM) and a self-attention module.

[0010] Step 5: Input the unified feature map into the decoder, and reconstruct a video frame sequence with the same input size through transpose convolution and upsampling operations;

[0011] Step 6: Based on the reconstruction loss between the reconstructed video frame sequence and the continuous video frame sequence, and the contrast loss obtained based on the momentum contrast learning mechanism, construct a hybrid loss function, optimize the parameters of the feature extraction network, the feature pyramid network FPN, the spatiotemporal fusion module and the decoder through backpropagation, complete the unsupervised pre-training, and save the encoder weights after training.

[0012] Step 7: In the supervised fine-tuning stage, obtain a continuous video frame sequence with binary classification labels as input, load some weights of the encoder, and freeze the parameters of the feature extraction network.

[0013] Step 8: The input labeled video frame sequence is processed by a loaded and frozen encoder to extract and fuse features, resulting in fused features;

[0014] Step 9: Input the fused features into the newly added classifier and map them to a binary classification probability distribution;

[0015] Step 10: Based on the cross-entropy loss between the binary classification probability distribution and the true label, optimize the parameters of the classifier and the unfrozen part of the encoder to complete supervised fine-tuning;

[0016] Step 11: Use the fine-tuned model to perform feature extraction and classification reasoning on the video frame sequence to be detected, and output a binary classification result indicating whether it is a frame insertion tampering.

[0017] According to a specific implementation of an embodiment of the present invention, the input tensor size of the unlabeled continuous video frame sequence is [B, 6, 3, 224, 224], where B is the batch size.

[0018] According to a specific implementation of an embodiment of the present invention, the shared feature extraction network uses ResNet50 pre-trained on the ImageNet dataset as the backbone network to independently extract three levels of feature maps for each frame of image.

[0019] According to a specific implementation of an embodiment of the present invention, step 3 specifically includes:

[0020] Step 3.1: Use 1×1 convolution to unify the features from different levels to the same number of channels to obtain lateral connectivity features;

[0021] Step 3.2: Using a top-down path, high-level features are upsampled and fused with adjacent low-level features level by level.

[0022] Step 3.3: Upsample the fused features of different sizes to a uniform size and stitch them together along the channel dimension;

[0023] Step 3.4: Aggregate the concatenated features using 3×3 convolution to obtain the fused feature map for each frame.

[0024] According to a specific implementation of an embodiment of the present invention, the channel-spatial attention module CBAM sequentially performs channel attention recalibration and spatial attention recalibration on the spliced ​​features;

[0025] The self-attention module is used to calculate the global correlation between feature channels to capture long-range dependencies and to perform residual connection with the output of CBAM.

[0026] According to a specific implementation of an embodiment of the present invention, the reconstruction loss is the sum of the mean squared error loss and the perceptual loss based on the feature extraction from the VGG network;

[0027] The contrastive loss is the InfoNCE loss constructed based on the momentum contrastive learning framework;

[0028] The hybrid loss function is a weighted sum of reconstruction loss and contrast loss.

[0029] According to one specific implementation of the present invention, the input tensor size of the continuous video frame sequence with binary classification labels is the same as the input size in the unsupervised pre-training stage.

[0030] According to a specific implementation of an embodiment of the present invention, the classifier includes a convolutional layer, a batch normalization layer, an activation function layer, a global average pooling layer, and a fully connected layer connected in sequence.

[0031] According to a specific implementation of an embodiment of the present invention, the unsupervised pre-training adopts the Adam optimizer and a cosine annealing learning rate scheduling strategy;

[0032] The supervised fine-tuning employs the Adam optimizer and a scheduling strategy that reduces the learning rate based on validation set performance.

[0033] According to a specific implementation of an embodiment of the present invention, the video frame data used in the unsupervised pre-training stage and the supervised fine-tuning stage are obtained by sampling and dividing the original video and the tampered video after being processed by video frame interpolation algorithms of different depths.

[0034] The video frame interpolation detection scheme based on unsupervised pre-training and multi-scale fusion in this embodiment of the invention includes: Step 1, obtaining an unlabeled continuous video frame sequence as input during the unsupervised pre-training stage; Step 2, inputting each frame in the video frame sequence into a shared feature extraction network to extract multi-scale visual features for each frame; Step 3, inputting the multi-scale visual features into a feature pyramid network (FPN) for cross-scale feature fusion to obtain a fused feature map for each frame; Step 4, concatenating the fused feature maps of multiple frames along the channel dimension and inputting them into a spatiotemporal fusion module to obtain a unified feature map that fuses spatial details and temporal dynamic information, wherein the spatiotemporal fusion module sequentially includes a channel-space attention module (CBAM) and a self-attention module; Step 5, inputting the unified feature map into a decoder, and reconstructing a video frame sequence with the same size as the input through transposed convolution and upsampling operations; Step 6, based on the reconstruction loss between the reconstructed video frame sequence and the continuous video frame sequence, and the base... Using the contrastive loss obtained from the momentum contrastive learning mechanism, a hybrid loss function is constructed. The parameters of the feature extraction network, feature pyramid network (FPN), spatiotemporal fusion module, and decoder are optimized through backpropagation to complete unsupervised pre-training, and the encoder weights are saved after training. Step 7: In the supervised fine-tuning stage, a continuous video frame sequence with binary classification labels is obtained as input, the encoder weights are loaded, and the parameters of the feature extraction network are frozen. Step 8: The input labeled video frame sequence is processed through the loaded and frozen encoder for feature extraction and fusion to obtain fused features. Step 9: The fused features are input into a newly added classifier and mapped to a binary classification probability distribution. Step 10: Based on the cross-entropy loss between the binary classification probability distribution and the true labels, the parameters of the classifier and the unfrozen parts of the encoder are optimized to complete supervised fine-tuning. Step 11: The fine-tuned model is used to perform feature extraction and classification inference on the video frame sequence to be detected, and the binary classification result indicating whether it is a frame insertion tampering is output.

[0035] The beneficial effects of this invention are as follows: By designing a two-stage training paradigm of "unsupervised pre-training - supervised fine-tuning" and deeply integrating multi-scale spatial feature extraction, local and global attention mechanisms, and joint reconstruction and contrastive learning objectives, a complete and efficient video frame interpolation detection system is constructed. This method reduces dependence on labeled data while achieving high expressive power for video spatiotemporal features and keen detection of hidden tampering traces. Therefore, it exhibits significant generalization advantages and stable high detection accuracy in actual tests across datasets and frame interpolation algorithms, providing a scalable and robust forensic solution for dealing with increasingly complex deepfake video tampering. Attached Figure Description

[0036] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 is a flowchart illustrating a video frame interpolation detection method based on unsupervised pre-training and multi-scale fusion provided by an embodiment of the present invention;

[0038] Figure 2 is a schematic diagram of the implementation process of a video frame interpolation detection method based on unsupervised pre-training and multi-scale fusion provided by an embodiment of the present invention;

[0039] Figure 3 is an architecture diagram of an unsupervised pre-training method provided by an embodiment of the present invention. Detailed Implementation

[0040] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0041] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0042] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this invention, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0043] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. The illustrations only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0044] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0045] This invention provides a video frame interpolation detection method based on unsupervised pre-training and multi-scale fusion, which can be applied to the process of detecting fake videos.

[0046] Referring to Figure 1, this is a flowchart illustrating a video frame interpolation detection method based on unsupervised pre-training and multi-scale fusion according to an embodiment of the present invention. As shown in Figures 1 and 2, the method mainly includes the following steps:

[0047] Step 1: In the unsupervised pre-training phase, obtain unlabeled continuous video frame sequences as input;

[0048] Step 2: Input each frame in the video frame sequence into a shared feature extraction network to extract multi-scale visual features for each frame;

[0049] Step 3: Input the multi-scale visual features into the Feature Pyramid Network (FPN) for cross-scale feature fusion to obtain the fused feature map for each frame.

[0050] Step 4: The fused feature maps of multiple frames are stitched together along the channel dimension and input into the spatiotemporal fusion module to obtain a unified feature map that fuses spatial details and temporal dynamic information. The spatiotemporal fusion module includes a channel-space attention module (CBAM) and a self-attention module.

[0051] Step 5: Input the unified feature map into the decoder, and reconstruct a video frame sequence with the same input size through transpose convolution and upsampling operations;

[0052] Step 6: Based on the reconstruction loss between the reconstructed video frame sequence and the continuous video frame sequence, and the contrast loss obtained based on the momentum contrast learning mechanism, construct a hybrid loss function, optimize the parameters of the feature extraction network, the feature pyramid network FPN, the spatiotemporal fusion module and the decoder through backpropagation, complete the unsupervised pre-training, and save the encoder weights after training.

[0053] Step 7: In the supervised fine-tuning stage, obtain a continuous video frame sequence with binary classification labels as input, load some weights of the encoder, and freeze the parameters of the feature extraction network.

[0054] Step 8: The input labeled video frame sequence is processed by a loaded and frozen encoder to extract and fuse features, resulting in fused features;

[0055] Step 9: Input the fused features into the newly added classifier and map them to a binary classification probability distribution;

[0056] Step 10: Based on the cross-entropy loss between the binary classification probability distribution and the true label, optimize the parameters of the classifier and the unfrozen part of the encoder to complete supervised fine-tuning;

[0057] Step 11: Use the fine-tuned model to perform feature extraction and classification reasoning on the video frame sequence to be detected, and output a binary classification result indicating whether it is a frame insertion tampering.

[0058] In practical implementation, Figure 1 shows a flowchart of a video frame interpolation detection method provided in this application; the video frame interpolation detection method provided in this application includes the following steps:

[0059] The input layer receives both unlabeled and labeled video frames. It first enters Stage 1, Unsupervised Pre-training: Through modality transformation and frame reconstruction tasks, the model learns the spatiotemporal features of video frame sequences from unlabeled data, generating pre-trained weights. Then, it enters Stage 2, Supervised Fine-tuning: The pre-trained fusion-reconstructed weights are loaded, and feature extraction and classification are performed on the labeled video. The model is further optimized using cross-entropy loss. Finally, the output layer outputs the binary classification result (0 or 1) of the video.

[0060] As shown in Figure 3, the input for pre-training is the original image sequence. For each frame in the sequence Apply the same random affine transformation (Cut, flip, rotate) to ensure The spatiotemporal correspondence is not disrupted. The image sequence is then fed into a shared feature extraction network.

[0061] The shared feature extractor uses a ResNet50 pre-trained on ImageNet as its backbone network to independently extract multi-scale features for each frame. The shared weight design is based on the following considerations: 1. All six frames share the same backbone network, with only 1 / 6 the number of parameters of an independent extractor. 2. It ensures that features from different frames reside in the same feature space, facilitating fusion. 3. It leverages large-scale pre-trained weights from ImageNet to accelerate convergence and improve performance.

[0062] For the input image ;

[0063] ;

[0064] Layer 1 output: Contains rich detail information and has high spatial resolution (56×56).

[0065] Layer 2 output: Balanced detail and semantic information, medium resolution (28×28);

[0066] Layer 3 output: Contains high-level semantic information, with a lower resolution (14×14).

[0067] ResNet50 structural details:

[0068] Initial layer: Conv1(7×7, stride=2) + BN + ReLU + MaxPool → Output feature map size is 1 / 4 of the input;

[0069] Layer 1: 3 Bottleneck blocks, 256 output channels, feature map size remains unchanged (56×56).

[0070] Layer 2: 4 Bottleneck blocks, 512 output channels, feature map size halved (28×28).

[0071] Layer 3: 6 Bottleneck blocks, 1024 output channels, feature map size halved (14×14).

[0072] The shared feature extractor outputs multi-scale feature pyramids C2, C3, and C4, containing low, medium, and high-level features for each frame. Feature pyramid fusion is then performed to integrate the multi-scale information.

[0073] FPN Construction Process: Step 1 - Lateral Connections:

[0074] First, a 1×1 convolution is used to unify the features from different levels to the same number of channels (hidden_channels=64): ;

[0075] Step 2 - Top-down path: Starting from the highest layer, upsample layer by layer and merge with the next layer: .

[0076] The upsampling process uses nearest neighbor interpolation, followed by element-wise addition.

[0077] Step 3 - Feature Alignment and Fusion: Upsample FPN features of different sizes to a uniform size (the size of P2, i.e., 56×56), and then concatenate them:

[0078] ;

[0079] Step 4 - Multi-scale feature aggregation:

[0080] The features from the three levels are concatenated along the channel dimension, and then fused through a convolutional layer:

[0081] ;

[0082] The fusion convolution uses 3×3 convolution + BatchNorm + ReLU to eliminate the aliasing effect caused by upsampling.

[0083] FPN preserves both detail (P2) and semantic (P4) information, while high-level semantic information enhances low-level features through a top-down path.

[0084] Temporal stitching: Restore the time dimension and stitch it to [B, 384, 56, 56]. CBAM: Channel attention M_c filters key feature channels, and spatial attention locates important regions. CBAM consists of two sub-modules: Channel Attention and Spatial Attention. Channel attention generates a channel attention map by aggregating spatial information, used to emphasize important feature channels. Specific calculation process:

[0085] Double pooling: Using both average pooling and max pooling simultaneously to capture different statistics: ;

[0086] Shared MLP: Processing two pooling results through a shared multilayer perceptron (MLP):

[0087] ;

[0088] in, , , This represents the compression ratio.

[0089] Element-level addition and activation: ;

[0090] Channel recalibration: ;

[0091] Spatial attention focuses on "where" is an important spatial location. The calculation process is as follows:

[0092] Channel-level pooling: Performs average pooling and max pooling along the channel dimension.

[0093] ;

[0094] Channel splicing and convolution: ;

[0095] Spatial recalibration: ;

[0096] Self-Attention: Calculates a global correlation matrix among 384 channels to capture long-range dependencies. Final Fusion: Convolution is compressed to 128 channels and upsampled back to 224.

[0097] Self-Attention is used to capture long-distance dependencies and global contextual information, making it suitable for handling temporal relationships across multiple frames of images.

[0098] Feature mapping: Generate Query, Key, and Value through three independent 1×1 convolutions:

[0099] ;

[0100] Spatial Dimension Reshaping: Reshape the feature map into a sequence for attention computation.

[0101] ;

[0102] Attention Calculation:

[0103] ;

[0104] Restoring spatial dimensions:

[0105] ;

[0106] Residual connectivity and learnable scaling:

[0107] ;

[0108] CBAM provides local, fine-grained attention, focusing on locally important regions (channel-level and spatial-level), while Self-Attention provides global, coarse-grained attention, capturing long-distance dependencies. First, CBAM performs coarse-grained filtering, then Self-Attention performs fine-grained association. The two complement each other, jointly improving the quality of feature representations.

[0109] Image reconstruction decoding: The fused features output by FPN are fed into the decoding network;

[0110] Network Architecture:

[0111] Block 1: Conv3x3 (128->64, s=1, p=1) ->BN ->ReLU.

[0112] Block 2: Conv3x3 (64->64, s=1, p=1)->BN->ReLU.

[0113] Block 3: Conv3x3 (64->64, s=1, p=1) ->BN ->ReLU.

[0114] Output Layer: Conv3x3 (64->18, s=1, p=1). Finally, it is mapped back to the original image.

[0115] Hybrid loss function:

[0116] Reconstruction loss: .

[0117] MoCo contrastive loss: The module includes a query encoder and a key encoder; the parameters of the key encoder are not updated through gradient backpropagation, but rather through the momentum moving average of the query encoder parameters, with the update formula as follows: This module maintains a negative sample queue with a length of 4096. It constructs the InfoNCE loss by calculating the similarity between the query sample and the positive sample key value, as well as the similarity between the query sample and the negative sample key value in the queue.

[0118] Dynamic weights are used for balancing, preventing any single loss term from dominating the gradient and ensuring that both reconstruction quality and feature discriminativeness are improved simultaneously. Feature decoupling: Reconstruction tasks focus on details, while contrastive tasks focus on semantic consistency; the two complement each other.

[0119] The monitoring and fine-tuning process is as follows:

[0120] Input: Image sequence: ,Label .

[0121] The shared feature extraction and multi-scale fusion parts are derived from the unsupervised pre-training in the first stage, with the parameters directly frozen within it. The output is a fused feature map. .

[0122] After obtaining the fused feature maps, the classifier head is used for training. The classifier head network consists of:

[0123] Conv Block 1: Conv3x3 (128->64, s=1, p=1) ->BN ->ReLU. Maintain dimensions. .

[0124] Conv Block 2: Conv3x3 (64->64, s=2, p=1) ->BN ->ReLU. Downsampling to .

[0125] Pooling: AdaptiveAvgPool2d((1, 1)). Will Compress to .

[0126] Flatten: Flatten into a vector .

[0127] Linear: Linear(64 -> 2). Maps to the category logits.

[0128] Supervision loss calculation: Softmax normalization is applied to logits to obtain the probability distribution. The cross-entropy between the predicted probability and the true label is calculated.

[0129] ;

[0130] in, It is the Softmax probability.

[0131] Experimental data: UCF101 was selected, which contains 400 videos. These 400 original videos were fed into a deep video frame rate upconversion method, such as the AdaCoF algorithm, to modify the video frame rate. Supervised and unsupervised training sets were divided in a 1:9 ratio, with 360 videos used for the unsupervised training set and 40 videos for the supervised training set. All labels were removed from the unsupervised training set, while the labels remained unchanged in the supervised training set. To improve model training time, the frame resolution was converted to 224×224.

[0132] Experimental Procedure: Six consecutive frames were used as a set of training data, totaling 40,662 data points, including 36,596 unsupervised training data points and 4,066 supervised training data points. To verify the effectiveness of the experiment, cross-dataset testing was conducted using the DANVIS dataset, and comparisons were made with other methods on different algorithms (RRIN, PerVFI, SGMVFI, Sepconv, etc.).

[0133] Training strategy:

[0134] Unsupervised pre-training:

[0135] Optimizer: Adam (lr=0.001, weight_decay=1e-4);

[0136] Scheduler: CosineAnnealingLR (T_max=40, eta_min=1e-6);

[0137] Epochs: 80;

[0138] Batch Size: 16.

[0139] Supervised fine-tuning:

[0140] Optimizer: Adam (lr=0.0001, weight_decay=1e-4);

[0141] Scheduler: ReduceLROnPlateau (factor=0.5, patience=5);

[0142] Epochs: 100;

[0143] Batch Size: 16;

[0144] Dropout: Applied before the fully connected layer in the classification header, p=0.5;

[0145] Weight decay: L2 regularization, coefficient 1e-4.

[0146] Table 1 shows the results of increasing the frame rate of the dataset from 25fps to 60fps, then training on the UCF101 dataset and testing on the DANVIS dataset. Other methods are all fully supervised training results. It can be seen that the performance across datasets is not good and none of them have achieved the desired results.

[0147] The proposed method outperforms other methods across datasets and also performs well on other frame interpolation methods, achieving good results. Table 2 shows the results after increasing the frame rate from 25fps to 90fps on the dataset, training on the UCF101 dataset, and testing on the DANVIS dataset. It can be seen that the performance from 25fps to 90fps is also good. This indicates that the frame rate has little impact on the detection results.

[0148] The results in Tables 1 and 2 show that because we used the UCF101 dataset and employed the ADACO method for frame interpolation, the test results on this dataset are optimal. However, the detection performance will decrease as the frame interpolation method changes. Table 3 indicates that the feature extraction effect selected by the method of this invention is optimal, achieving relatively ideal results.

[0149] Table 1

[0150]

[0151] Table 2

[0152]

[0153] Table 3 shows that the feature extraction effect selected by the method of the present invention is optimal, and relatively ideal results have been achieved.

[0154] Table 3

[0155]

[0156] This embodiment presents a video frame interpolation detection method based on unsupervised pre-training and multi-scale fusion. By designing a two-stage training paradigm of "unsupervised pre-training - supervised fine-tuning," and deeply integrating multi-scale spatial feature extraction, local and global attention mechanisms, and joint reconstruction and contrastive learning objectives, a complete and efficient video frame interpolation detection system is constructed. This method reduces dependence on labeled data while achieving high expressive power for video spatiotemporal features and keen detection of hidden tampering traces. Therefore, it demonstrates significant generalization advantages and stable high detection accuracy in practical tests across datasets and frame interpolation algorithms, providing a scalable and robust forensic solution for dealing with increasingly complex deepfake video tampering.

[0157] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof.

[0158] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A video frame interpolation detection method based on unsupervised pre-training and multi-scale fusion, characterized in that, include: Step 1: Obtain an unlabeled continuous video frame sequence as input during the unsupervised pre-training stage. Step 2: Input each frame in the video frame sequence into a shared feature extraction network to extract multi-scale visual features for each frame. Step 3: Input the multi-scale visual features into a Feature Pyramid Network (FPN) for cross-scale feature fusion to obtain a fused feature map for each frame. Step 4: Concatenate the fused feature maps of multiple frames along the channel dimension and input them into a spatiotemporal fusion module to obtain a unified feature map that integrates spatial details and temporal dynamic information. The spatiotemporal fusion module includes a Channel-Spatial Attention (CBAM) module and a self-attention module. Step 5: Input the unified feature map into a decoder and perform transposed convolution and upsampling operations. Step 6: Reconstruct a video frame sequence with the same size as the input; Based on the reconstruction loss between the reconstructed video frame sequence and the continuous video frame sequence, and the contrast loss obtained based on the momentum contrastive learning mechanism, construct a hybrid loss function, optimize the parameters of the feature extraction network, the feature pyramid network FPN, the spatiotemporal fusion module and the decoder through backpropagation, complete unsupervised pre-training, and save the encoder weights after training; Step 7: In the supervised fine-tuning stage, obtain a continuous video frame sequence with binary classification labels as input, load the encoder weights, and freeze the parameters of the feature extraction network; Step 8: Extract and fuse the input labeled video frame sequence through the loaded and frozen encoder to obtain fused features; Step 9: Input the fused features into the newly added classifier and map them to a binary classification probability distribution; Step 10: Based on the cross-entropy loss between the binary classification probability distribution and the true label, optimize the parameters of the classifier and the unfrozen part of the encoder to complete supervised fine-tuning; Step 11: Use the fine-tuned model to perform feature extraction and classification inference on the video frame sequence to be detected, and output a binary classification result indicating whether it is a frame insertion tampering.

2. The method according to claim 1, characterized in that, The input tensor size of the unlabeled continuous video frame sequence is [B, 6, 3, 224, 224], where B is the batch size.

3. The method according to claim 1, characterized in that, The shared feature extraction network uses ResNet50 pre-trained on the ImageNet dataset as the backbone network to independently extract three levels of feature maps for each frame of image.

4. The method according to claim 3, characterized in that, Step 3 specifically includes: Step 3.1, unifying features from different levels to the same number of channels using 1×1 convolution to obtain lateral connection features; Step 3.2, upsampling high-level features and fusing them with adjacent low-level features level by level using a top-down path; Step 3.3, upsampling the fused features of different sizes to a uniform size and stitching them together in the channel dimension; Step 3.4, aggregating the stitched features using 3×3 convolution to obtain the fused feature map for each frame.

5. The method according to claim 1, characterized in that, The channel-space attention module CBAM sequentially performs channel attention recalibration and spatial attention recalibration on the concatenated features; the self-attention module is used to calculate the global correlation between feature channels to capture long-distance dependencies and performs residual connection with the output of CBAM.

6. The method according to claim 1, characterized in that, The reconstruction loss is the sum of the mean squared error loss and the perceptual loss based on feature extraction from the VGG network; the contrast loss is the InfoNCE loss constructed based on the momentum contrastive learning framework; and the hybrid loss function is the weighted sum of the reconstruction loss and the contrast loss.

7. The method according to claim 1, characterized in that, The input tensor size of the continuous video frame sequence with binary classification labels is the same as the input size in the unsupervised pre-training stage.

8. The method according to claim 1, characterized in that, The classifier comprises a convolutional layer, a batch normalization layer, an activation function layer, a global average pooling layer, and a fully connected layer connected in sequence.

9. The method according to claim 1, characterized in that, The unsupervised pre-training employs the Adam optimizer and a cosine annealing learning rate scheduling strategy; the supervised fine-tuning employs the Adam optimizer and a scheduling strategy that reduces the learning rate based on the performance of the validation set.

10. The method according to any one of claims 1 to 9, characterized in that, The video frame data used in the unsupervised pre-training stage and the supervised fine-tuning stage are obtained by sampling and dividing the original video and the tampered video after being processed by video frame interpolation algorithms of different depths.

Images