Deep video inpainting detection method and system based on ConvNeXt

Abstract

The application discloses a deep video repair detection method and system based on ConvNeXt, which first performs frame processing on videos in a data set to obtain a video frame sequence. Secondly, a video repair detection network containing a time sequence residual extraction module, a filtering residual extraction module, a ConvNeXt double-flow feature extraction module, a feature fusion module and a pyramid structure feature recovery module is constructed. Then, a loss function and a parameter optimization algorithm are constructed. Finally, the trained video repair detection network is tested, and the network performance is evaluated. The system includes a video processing module, a time sequence feature extraction module, a filtering feature extraction module, a feature fusion module, a pyramid structure feature recovery module and a detection result generation module. The detection result has high accuracy and integrity, and the high performance of the video repair detection network is ensured.

Description

Technical Field

[0001] This invention relates to the field of digital video tampering detection technology, specifically to a deep video restoration tampering detection method and system based on ConvNeXt. Background Technology

[0002] Video restoration is a common video editing technique that aims to repair and fill in missing or damaged areas in a video by using visually plausible pixels, ensuring the consistency of the video content in time and space, and giving the restored video a natural visual effect. With the widespread application of deep learning technology in video processing, the effectiveness of video restoration has been improved unprecedentedly, making it almost impossible to distinguish the authenticity of video content with the naked eye. Therefore, deep video restoration tamper detection has become a very important research topic in multimedia content analysis and information security.

[0003] Most current deep video restoration and tamper detection methods break down videos into a series of consecutive frames and then process them using image restoration detection techniques. However, due to the lack of temporal correlation in the analysis process, the models cannot learn effective features, resulting in poor performance. A few video-based methods use Hessian matrices and Local Binary Pattern (LBP) coherence analysis to assist detection. However, their performance drops significantly or even fails when faced with deep learning-based video restoration and tamper detection techniques or video data with small restoration areas and significant style differences. This is because deep video restoration and tamper detection techniques can acquire information from adjacent frames, introducing traces different from traditional restoration techniques. Therefore, it is necessary to provide a new deep video restoration and tamper detection method and system to solve the above problems. Summary of the Invention

[0004] The purpose of this invention is to at least address the technical problems existing in the prior art. To this end, this invention proposes a deep video restoration detection method and system based on ConvNeXt, which combines multi-scale features of video in the temporal and spatial domains to improve the accuracy, completeness, and generalization ability of the model in detecting target video restoration tampering.

[0005] To achieve the above objectives, a first aspect of the present invention provides a depth video inpainting detection method based on ConvNeXt, characterized by comprising the following steps:

[0006] S1: First, the videos in the dataset are segmented into frames. The dataset contains video sequences V = {V...} i |i=1,2,…,N}, where N represents the number of videos. Each video is processed into frames to obtain a video frame sequence I={I i|i=1,2,…,n}, where n represents the number of video frames. Then, all video frames are divided into training, validation, and test sets, with each video as a unit.

[0007] S2: Construct a deep video inpainting detection network based on ConvNeXt. The video inpainting detection network includes a temporal residual extraction module, a filter residual extraction module, a ConvNeXt dual-stream feature extraction module, a feature fusion module, and a pyramid structure feature recovery module.

[0008] S3: Construct the loss function and parameter optimization algorithm used for training the video restoration detection network. Train the video restoration detection network using video frames from the training and validation sets. The video restoration detection network processes an odd number of video frames in each time window. After training, save the network model and the optimal weight parameters.

[0009] S4: Test the trained video inpainting detection network by taking video frames from the test set as input, outputting the detection mask, and evaluating the performance of the video inpainting detection network.

[0010] As a preferred technical solution, in step S2, the temporal residual extraction module includes a three-dimensional (3D) convolutional layer and a spatial rich model (SRM) filtering layer. The input is first processed by 3D convolution, then by SRM filtering, and the output is the temporal residual feature F. t .

[0011] As a preferred technical solution, in step S2, the filter residual extraction module includes a Laplacian of Gaussian (LoG) filter layer and a Laplacian filter layer. The input is first processed by LoG filtering, then by Laplacian filtering, and the output is the filter residual feature F. p .

[0012] As a preferred technical solution, in step S2, the ConvNeXt dual-stream feature extraction module includes two single-stream temporal residual feature extraction networks, each of which consists of four-stage ConvNeXt blocks. The input to the first single-stream temporal residual feature extraction network is the temporal residual feature F. t The input to the second single-stream temporal residual feature extraction network is the filtered residual feature F. p After feature extraction through four stages of ConvNeXt blocks, the first multi-level temporal residual feature is output. and the second multi-level filtering residual feature

[0013] As a preferred technical solution, in step S2, the feature fusion module is input with the first multi-level temporal residual feature F. 1 and the second multi-level filtering residual feature F 2 The temporal residual features and filtered residual features at the same level are added point by point to output a multi-level fused feature F = {F i |i=1,2,3,4}.

[0014] As a preferred technical solution, in step S2, the pyramid structure feature recovery module includes a pyramid feature abstraction structure (PPM), a cross-level feature upsampling fusion structure, and a feature recovery structure. The input is a multi-level fusion feature F. The high-level feature F4 is first processed through the PPM structure to obtain the abstract feature F′4. Then, F1, F2, F3, and F′4 are fed into the cross-level feature upsampling fusion structure. In the cross-level feature upsampling fusion structure, F1, F2, and F3 are first convolved to obtain F′1, F′2, and F′3. Then, F′4 is upsampled and fused with F′3 to obtain F″3. F″3 is then upsampled and fused with F′2 to obtain F″2. Finally, F″2 is upsampled and fused with F′1 to obtain F″1. In the feature recovery structure, F″2, F″3, and F′4 are upsampled and concatenated with F″1 along the channel dimension, and a detection mask is obtained through a deconvolution layer and a softmax layer.

[0015] As a preferred technical solution, in step S3, the loss function used for training the video restoration detection network includes the Dice loss function and the IoU loss function, which together form a hybrid loss function, specifically expressed as follows:

[0016]

[0017]

[0018]

[0019] Where N represents the total number of pixels in the prediction result, and y represents the pixel value of the label mask. ε represents the pixel value of the predicted mask, j represents the j-th pixel, ε is the smoothing coefficient to prevent the denominator from being 0, and the weights λ1 and λ2 are both set to 0.5.

[0020] In a second aspect, the present invention provides a deep video restoration and detection system based on ConvNeXt, comprising a video processing module, a temporal feature extraction module, a filtered feature extraction module, a feature fusion module, a pyramid structure feature restoration module, and a detection result generation module.

[0021] The video processing module is used to divide the video into consecutive video frames according to the frame rate and save them in a folder as images.

[0022] The temporal feature extraction module is located alongside the filtering feature extraction module. Given a fixed sequence of video frames, it first utilizes 3D convolution and SRM filtering, then employs a pre-trained ConvNeXt feature extraction network to output multi-level temporal residual features.

[0023] The filtering feature extraction module is located alongside the temporal feature extraction module. Given a fixed sequence of video frames, it first utilizes LoG filtering and Laplacian filtering, then employs a pre-trained ConvNeXt feature extraction network to output multi-level filtering residual features.

[0024] The feature fusion module takes multi-level temporal residual features and multi-level filtered residual features as input, fuses the temporal residual features and filtered residual features at the same level, and outputs multi-level fused features.

[0025] The pyramid structure feature recovery module takes multi-level fusion features as input, uses a pre-trained pyramid feature abstraction structure, cross-level feature upsampling fusion structure and feature recovery structure to determine whether the video frame to be detected has been repaired, outputs the detection mask of the corresponding video frame, and saves it in a folder as an image.

[0026] The detection result generation module combines the detection masks corresponding to the video frames into a single video to obtain the final detection result.

[0027] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0028] (1) This invention extracts abrupt change information of image pixel values ​​through LoG and Lapacian filtering to obtain the spatial residual features. Motion residual information in the temporal dimension is extracted through 3D convolution and SRM filtering to obtain the temporal residual features. A ConvNeXt dual-stream feature extraction module is designed to fully exploit the information contained in the filtering residual features and temporal residual features. Furthermore, a feature fusion module is used to merge the spatiotemporal features, utilizing the spatial features and noise features of a single video frame, as well as the temporal features of multiple video frames, to detect video restoration. The detection results have high accuracy and completeness.

[0029] (2) This invention fully explores the connection between features at different levels. The high-level features generated by the deep network have strong semantic representation capabilities, while the low-level features generated by the shallow network have strong geometric detail representation capabilities. The pyramid structure feature recovery module uses a cross-level feature upsampling fusion structure to fully combine the advantages of the two features, which greatly helps the detection network learn the characteristics of video repair operations and ensures the high performance of the video repair detection network.

[0030] (3) This invention uses a hybrid loss function to train the detection network. Since the video restoration area usually only occupies a small part of the entire video frame, there is a strong imbalance between positive and negative samples. Therefore, L... Dice This can be solved very well. At the same time, using L... IoU This can encourage the output mask of the detection network to move closer to the label, thereby improving the accuracy of the detection network. Attached Figure Description

[0031] Figure 1 This is a diagram showing the overall structure of the ConvNeXt-based deep video inpainting detection method of the present invention.

[0032] Figure 2 This is a flowchart of the depth video restoration and detection method based on ConvNeXt of the present invention;

[0033] Figure 3 Here is a diagram of the SRM filter kernel parameters;

[0034] Figure 4(a) shows the parameters of the LoG filter kernel;

[0035] Figure 4(b) shows the parameters of the Lapacian filter kernel;

[0036] Figure 5 Here is a diagram of the network structure of ConvNeXt-T;

[0037] Figure 6(a) shows the original video frames of the test sample;

[0038] Figure 6(b) shows the video frames of the test sample after repair;

[0039] Figure 6(c) shows the label mask of the test sample;

[0040] Figure 6(d) shows the filter residuals of the test samples;

[0041] Figure 7 This is a schematic diagram of the system of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0043] Example 1:

[0044] This embodiment provides a deep video inpainting detection method based on ConvNeXt. The video inpainting detection network structure is as follows: Figure 1As shown, fixed-sequence video frames are input side-by-side into the temporal residual extraction module and the filter residual extraction module to obtain temporal residuals and filter residuals. Then, a single-stream temporal residual feature extraction network and a single-stream filter residual extraction network simultaneously extract features from the temporal residuals and filter residuals, resulting in multi-level temporal residual features and multi-level filter residual features. Next, a feature fusion module fuses the temporal residual features and filter residual features at the same level to obtain multi-level fused features. Finally, a pyramid structure feature recovery module first abstracts the high-level features, then performs cross-level feature upsampling and fusion, and finally restores the features to the input video frame size to obtain the final detection mask.

[0045] This embodiment utilizes three different video restoration algorithms to create the dataset required for restoration detection based on the YouTubeVOS dataset. The experiment was conducted on an Ubuntu system using Python version 3.8 and PyTorch artificial neural network library version 1.8.0, CUDA version 11.2, and a single NVIDIA 3090 graphics card.

[0046] This embodiment provides a depth video inpainting detection method based on ConvNeXt, the process of which is as follows: Figure 2 As shown, it includes the following steps:

[0047] Step S1: First, perform frame segmentation on the videos within the dataset. The dataset contains video sequences V = {V...} i |i=1,2,…,N}, where N represents the number of videos. Each video is processed into frames to obtain a video frame sequence I={I i |i=1,2,…,n}, where n represents the number of video frames. Then, all video frames are divided into training, validation, and test sets, with each video as a unit.

[0048] Specifically, the dataset includes both unrepaired and repaired videos, totaling 2000 videos. They are divided in an 8:1:1 ratio: 1600 videos are used for the training set, 200 videos for the validation set, and 200 videos for the test set.

[0049] Step S2: Construct a deep video inpainting detection network based on ConvNeXt. The video inpainting detection network includes a temporal residual extraction module, a filter residual extraction module, a ConvNeXt dual-stream feature extraction module, a feature fusion module, and a pyramid structure feature recovery module.

[0050] Specifically, the temporal residual extraction module consists of a 3D convolutional layer and an SRM filter layer. Since video data is being detected, the content between adjacent video frames exhibits continuity in the temporal dimension. Using 3D convolution allows for the aggregation of video frames in the temporal dimension, extracting residual information about object motion over time. SRM is a high-dimensional spatial steganalysis image feature analysis technique, widely used in image tampering detection to analyze whether the correlation between neighborhood pixels has been disrupted. The filter structure of the SRM filter layer is as follows: Figure 3 As shown, similar to a convolution operation, the three filters correspond to the three RGB channels respectively, and are used to enhance the repair traces.

[0051] Specifically, the filter residual extraction module consists of a LoG filter layer and a Laplacian filter layer. Laplacian filtering is a sharpening filter, representing the second derivative matrix of image pixels and containing abrupt changes in pixel values. It is a commonly used filtering algorithm for edge extraction, and its filter kernel is shown in Figure 4(a). Since video restoration often results in blurred edges in the restored area, Laplacian filtering can highlight the edge information of the restored area, providing valuable assistance for the model to learn restoration features. To prevent the noise in the image from becoming more pronounced when calculating the second derivative using Laplacian filtering, LoG filtering is first used to smooth the image, and then Laplacian filtering is used to enhance the edges of the restored area. The LoG filter kernel is shown in Figure 4(b).

[0052] Specifically, the ConvNeXt dual-stream feature extraction module consists of a single-stream temporal residual feature extraction network and a single-stream filtered residual feature extraction network arranged in parallel, both employing the ConvNeXt-T network structure, such as... Figure 5 As shown.

[0053] Specifically, the pyramid structure feature recovery module consists of a pyramid feature abstraction structure (PPM), a cross-level feature upsampling fusion structure, and a feature recovery structure. The structure of the PPM is as follows: Figure 1As shown, the network consists of pooling layers of different scales, convolutional layers (including normalization and activation functions), and upsampling layers. The purpose is to aggregate contextual information from different regions and improve the network's ability to acquire global information. Pooling at different scales is applied to high-level features to obtain multiple feature maps of different sizes. These features are then concatenated along the channel dimension to obtain high-level abstract features. The cross-level upsampling feature fusion structure sequentially fills upsampled features with the previous level's features using bilinear interpolation, from high to low, thus achieving a balance between global semantic information and local detail information. The feature recovery structure processes the fused features through deconvolutional layers, normalization, and activation functions, then passes them through a softmax layer to calculate probabilities. Based on these probabilities, it determines whether a pixel has been repaired, resulting in a detection mask.

[0054] Step S3: Construct the loss function and parameter optimization algorithm for training the video restoration detection network. Train the video restoration detection network using video frames from the training and validation sets. The video restoration detection network processes an odd number of video frames in each time window. After training, save the network model and the optimal weight parameters.

[0055] Specifically, the loss function used in training the video inpainting detection network is a hybrid loss function combining the Dice loss function and the IoU loss function. The Dice loss function, derived from the Dice coefficient, is a metric used to evaluate the similarity between two samples. It mitigates the negative impact of an imbalance between positive and negative samples, a problem addressed since the area of ​​the video inpainting region typically occupies only a small portion of the video frame. The IoU loss function promotes a larger intersection between the detection result mask and the corresponding label, while also predicting more accurate region edges. Using this hybrid loss function improves the accuracy and completeness of the model's detection results.

[0056] Step S4: Test the trained video inpainting detection network by taking video frames from the test set as input, outputting the detection mask, and evaluating the performance of the video inpainting detection network.

[0057] This embodiment trains the model on the YouTubeVOS dataset restored by the FuseFormer and E2FGVI video restoration algorithms. The in-database (YouTubeVOS) and cross-database (DAVIS) test results are shown in Table 1 below. The annotations after the dataset indicate the video restoration algorithms used. In the experiment, Figure 6(a) shows the original video frames selected for the test samples, Figure 6(b) shows the video frames after restoration for the test samples, Figure 6(c) shows the label mask obtained for the test samples, and Figure 6(d) shows the filtering residuals for the test samples.

[0058] Table 1. Test results of the training model on the YouTubeVOS (FuseFormer, E2FGVI) dataset.

[0059]

[0060] As shown in Table 1, the model trained on the YouTubeVOS (FuseFormer, E2FGVI) dataset in this embodiment achieves good performance in the in-database test. Due to the significant difference in video styles between the DAVIS dataset and the YouTubeVOS dataset, the model's performance in the cross-database test is somewhat affected, but it still outperforms other existing video restoration and detection methods, verifying the effectiveness of this invention.

[0061] Example 2:

[0062] This embodiment also provides a deep video inpainting detection system based on ConvNeXt, such as... Figure 7 As shown, it includes a video processing module, a temporal feature extraction module, a filtering feature extraction module, a feature fusion module, a pyramid structure feature recovery module, and a detection result generation module.

[0063] The video processing module is used to divide the video into consecutive video frames according to the frame rate and save them in a folder as images.

[0064] The temporal feature extraction module is located alongside the filtering feature extraction module. Given a fixed sequence of video frames, it first uses 3D convolution and SRM filtering to extract the temporal residuals between video frames. Then, it uses a pre-trained ConvNeXt feature extraction network to output multi-level temporal residual features.

[0065] The filter feature extraction module is located alongside the temporal feature extraction module. Given a fixed sequence of video frames, LoG filtering and Laplacian filtering are first used to extract the filter residuals of the video frames. Then, a pre-trained ConvNeXt feature extraction network is used to output multi-level filter residual features.

[0066] The feature fusion module takes multi-level temporal residual features and multi-level filtered residual features as input, fuses the temporal residual features and filtered residual features at the same level, and outputs multi-level fused features.

[0067] The pyramid structure feature recovery module takes multi-level fused features as input, uses a pre-trained pyramid feature abstraction structure, cross-level feature upsampling fusion structure and feature recovery structure to calculate the probability that each pixel in the video frame is the original image and the probability that it is the repaired image, determines whether the video frame to be detected has been repaired, outputs the detection mask of the corresponding video frame, and saves it in a folder as an image.

[0068] The detection result generation module combines the detection masks corresponding to the video frames into a single video to obtain the final detection result.

[0069] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A depth video inpainting detection method based on ConvNeXt, characterized in that, Includes the following steps: S1: Perform frame segmentation on the videos within the dataset, which contains video sequences. ,in This indicates the number of videos, each of which is processed into a video frame sequence. , where n represents the number of video frames; Then, all video frames are divided into training, validation, and test sets, with each video as a unit. S2: Construct a deep video inpainting detection network based on ConvNeXt. The video inpainting detection network includes a temporal residual extraction module, a filter residual extraction module, a ConvNeXt dual-stream feature extraction module, a feature fusion module, and a pyramid structure feature recovery module. The temporal residual extraction module includes a 3D convolutional layer and a spatial rich model SRM filter layer. The input is first processed by 3D convolution, then by SRM filtering, and the output is temporal residual features. ; The filter residual extraction module includes a Gaussian-Laplace (LoG) filter layer and a Laplacian (Laplacian) filter layer. The input is first processed by the LoG filter, then by the Laplacian filter, and the output is the filter residual feature. ; The ConvNeXt dual-stream feature extraction module includes two single-stream temporal residual feature extraction networks, each consisting of four-stage ConvNeXt blocks; the input to the first single-stream temporal residual feature extraction network is the temporal residual features. The input to the second single-stream temporal residual feature extraction network is the filtered residual feature. After feature extraction through four stages of ConvNeXt blocks, the first multi-level temporal residual feature is output. and the second multi-level filtering residual feature ; The feature fusion module takes the first multi-level temporal residual feature as input. and the second multi-level filtering residual feature The temporal residual features and filtered residual features at the same level are added point by point to output a multi-level fused feature. ; The pyramid structure feature recovery module includes a pyramid feature abstraction structure (PPM), a cross-level feature upsampling fusion structure, and a feature recovery structure; the input is a multi-level fused feature. High-level features First, abstract features are obtained through the PPM structure. Then , , , The data is fed into a cross-level feature upsampling fusion structure; in the cross-level feature upsampling fusion structure, the data is first processed... , , Convolution is performed to obtain , , Then After upsampling and Fusion Then After upsampling and Fusion Finally After upsampling and Fusion ; In the feature recovery structure, , , After upsampling and The detection mask is obtained by stitching along the channel dimension and passing it through a deconvolution layer and a softmax layer. S3: Construct the loss function and parameter optimization algorithm used for training the video restoration detection network. Train the video restoration detection network using video frames from the training and validation sets. The video restoration detection network processes an odd number of video frames in each time window. After training, save the network model and the best weight parameters. S4: Test the trained video inpainting detection network by taking video frames from the test set as input, outputting the detection mask, and evaluating the performance of the video inpainting detection network.

2. The depth video inpainting detection method based on ConvNeXt according to claim 1, characterized in that, In step S3, the loss function includes the Dice loss function and the IoU loss function, which together form a hybrid loss function, specifically expressed as follows: in, This indicates the total number of pixels in the prediction result. This represents the pixel value of the label mask. This represents the pixel value of the predicted mask. Indicates the first 1 pixel, It is the smoothing coefficient. and For weights.

3. A ConvNeXt-based deep video inpainting detection system, used to implement the method described in any one of claims 1 to 2, characterized in that, It includes a video processing module, a temporal feature extraction module, a filtered feature extraction module, a feature fusion module, a pyramid structure feature recovery module, and a detection result generation module; The video processing module is used to divide the video into consecutive video frames according to the frame rate and save them; The temporal feature extraction module takes a fixed sequence of video frames as input and uses 3D convolution, SRM filtering, and a pre-trained ConvNeXt feature extraction network to output multi-level temporal residual features. The filter feature extraction module takes a fixed sequence of video frames as input, and uses LoG filtering, Laplacian filtering, and a pre-trained ConvNeXt feature extraction network to output multi-level filter residual features. The feature fusion module takes multi-level temporal residual features and multi-level filtered residual features as input, fuses the temporal residual features and filtered residual features at the same level, and outputs multi-level fused features. The pyramid structure feature recovery module takes multi-level fused features as input, uses a pre-trained pyramid feature abstraction structure, cross-level feature upsampling fusion structure and feature recovery structure to determine whether the video frame to be detected has been repaired, and outputs the detection mask of the corresponding video frame.

Images