Free-form video inpainting method based on multi-dimensional information aggregation and pixel-level adversarial learning

Abstract

This invention relates to the fields of deep neural networks, video inpainting, and generative adversarial learning in image machine learning and computer vision research. It presents a free-format video inpainting method based on multi-dimensional information aggregation and pixel-level adversarial learning, comprising a multi-dimensional cross-correlation video inpainting model and a pixel-level adversarial learning method, which are connected through reconstructed frames. By using multi-dimensional cross-correlation enhancement technology, contextual information in different dimensions is aggregated, and dimensional decomposition of the multi-dimensional cross-correlation enhancement is performed to reduce computational complexity. A pixel-level adversarial learning method is used to judge the quality of the inpainted video, enhancing the targeted optimization of the generator. This free-format video inpainting method integrates multi-dimensional cross-correlation enhancement and pixel-level adversarial learning, fully considering the differences in different contextual information within the video, preserving temporal and spatial consistency in the reconstructed video, and making the inpainted video more realistic.

Description

Technical Field

[0001] This invention relates to the fields of deep neural networks, video restoration, and generative adversarial learning in machine learning and computer vision research, specifically to a free-format video restoration method based on multi-dimensional information aggregation and pixel-level adversarial learning. Background Technology

[0002] Video inpainting aims to fill in missing regions of a video with appropriate content while maintaining both visual and semantic consistency. Video inpainting has many traditional applications, such as removing objects from videos, repairing damaged video materials, automatic content filtering, and visual privacy filtering. In addition, video inpainting can be combined with augmented reality (AR) to remove existing objects and add new elements to increase scene diversity. Unlike traditional video inpainting tasks where missing regions are static and connected, the missing regions in free-form video inpainting are neither fixed in shape nor position. Work based on convolutional network variants focuses on collecting and fusing information from the valid remaining parts of the video. By applying pseudomasks in deep networks, these methods can successfully distinguish between valid and invalid features. However, because the convolutional kernel remains unchanged across the entire feature map during a single convolutional operation, it cannot adaptively and specifically collect contextual information based on different locations and different missing information. This often results in videos with blurred watermarks and boundary artifacts.

[0003] Optical flow-based methods and feature block matching-based methods are two commonly used approaches in video restoration. The former first completes the video's optical flow and then uses it to aid in video restoration. Its limitation lies in the fact that the restoration result heavily relies on the accuracy of optical flow prediction and completion, and it overemphasizes temporal information collection while neglecting effective spatial information. The latter searches for similar image blocks in the spatial dimension to complete the damaged area, but currently this method is only applicable to a single dimension because its computational complexity becomes too high when extended to high-dimensional scenes.

[0004] On the other hand, recent works have attempted to utilize Generative Adversarial Networks (GANs) to maintain visual consistency, semantic plausibility, and temporal consistency in reconstructed videos. However, existing GANs often fail to achieve good results in free-format video inpainting. Firstly, the occlusion ratios between videos in free-format video inpainting vary greatly, causing the discriminator to be unable to effectively identify the quality of reconstructed videos under different occlusion ratios. Secondly, existing GAN discriminators often only evaluate the reconstructed video as a whole, failing to provide fine-grained evaluations of specific regions or even individual pixels. This results in a lack of specificity in optimizing the generator using the discriminator, leading to slow convergence during generator training and limited optimization effects.

[0005] Therefore, it is necessary to further improve the free-format video restoration method based on multi-dimensional information aggregation and pixel-level adversarial learning to solve the above problems. Summary of the Invention

[0006] The purpose of this application is to address and overcome the shortcomings of existing technologies and applications, and to provide a free-format video restoration method based on multi-dimensional information aggregation and pixel-level adversarial learning. This method effectively solves the problem of not being able to adaptively collect contextual information based on different locations and missing information in video restoration, thereby preserving temporal and spatial consistency in the reconstructed video and making the restored video more realistic.

[0007] The purpose of this application is achieved through the following technical solution: a free-format video restoration method based on multidimensional information aggregation and pixel-level adversarial learning, including a multidimensional cross-correlation video restoration model used as a generator and an occlusion discriminator in the pixel-level adversarial learning method, wherein the multidimensional cross-correlation video restoration model used as a generator and the occlusion discriminator in the pixel-level adversarial learning method are associated through a regeneration method.

[0008] The multidimensional cross-correlation video restoration model used as the generator consists of three parts: upsampling, dilation, and pre- and post-downsampling. Multidimensional cross-correlation modules are inserted before and after the dilation module. The multidimensional cross-correlation module collects information from a given D-dimensional target feature map and a reference feature map in the same feature space, calculates the cross-correlation between the target feature map and the reference feature map, assigns corresponding cross-correlation weights based on the cross-correlation of the feature blocks, applies the cross-correlation weights to the center values ​​of the corresponding reference feature blocks, and obtains the final cross-correlation enhancement result through weighted summation. The computational complexity of cross-correlation between dimensions is reduced by reducing the dimensionality of the feature map.

[0009] The occlusion discriminator in the pixel-level adversarial learning method consists of a high-pass filter and a mask generator connected in series. The high-pass filter and the mask generator output a mask image that infers whether the image is real or fake. The occlusion discriminator evaluates the pixels in the reconstructed video and distinguishes the pixels generated by the mask generator from the original preserved pixels.

[0010] Preferably, when the multidimensional cross-correlation video restoration model is used as a generator, downsampling is achieved by (2+1)D gated convolution combined with residual mechanism, and upsampling is achieved by (2+1)D gated convolution combined with residual mechanism plus nearest neighbor interpolation.

[0011] Preferably, the residual mechanism (2+1)D is combined with the residual mechanism of (2+1)D gated convolution kernel. The (2+1)D gated convolution combined with the residual mechanism serves as the basic convolution module of the multidimensional cross-correlation video restoration model, which is used to reduce the number of parameters in the multidimensional cross-correlation video restoration model.

[0012] Preferably, the expansion module adaptively and effectively aggregates the contextual information obtained from upsampling and downsampling in the reference feature map by employing a multidimensional cross-correlation module, and expands the receptive field using dilated convolution.

[0013] Preferably, the target feature block With target feature map Chinese-Israeli index Centered on, reference feature block With reference feature map Chinese-Israeli index Centered on this, we obtain the formula for calculating cross-correlation:

[0014] ;

[0015] and Given a D-dimensional target feature map and a reference feature map, where yes The index is eigenvalues, yes The index is eigenvalues, It's a similarity calculation; the shape of the feature block is... , by each dimension The corresponding feature block radius Decide.

[0016] Preferably, assigning cross-correlation weights refers to applying them to the reference feature map. In China, with The radius of the receptive field centered on the center is expressed as Apply step size in each dimension Corresponding index The set is represented as ,in ; Obtain the cross-correlation results graph Defined as all in the target feature map and reference feature map The cross-correlation between feature blocks in the data, i.e. The cross-correlation results are plotted. use The function is applied to obtain the cross-correlation weights. The formula is:

[0017] ;

[0018] Used for updating Cross-correlation calculation results The result is obtained through the following calculations:

[0019]

[0020] The entire process described above can be simplified as follows: .

[0021] Preferably, the D-dimensional target feature map has four dimensions: height, width, channel, and time. High dimensionality leads to computational complexity; therefore, dimensionality reduction is used to address the computational difficulty, and the calculation is as follows:

[0022]

[0023] Dimensional decomposition breaks down the 4-dimensional cross-correlation calculation into a 3-dimensional (space-time) cross-correlation calculation and a 2-dimensional (time-channel) cross-correlation calculation.

[0024] Preferably, the high-pass filter is used to highlight the high-frequency components in free-format video restoration, which can help the discriminator identify restoration traces in the video.

[0025] Preferably, the mask generator is connected after the high-pass filter to generate the final output of the pixel-level adversarial learning discriminator for free-format video restoration.

[0026] Preferably, the mask image is used to distinguish between the repaired portion and the natural portion of the reconstructed frame; the mask generator loss function

[0027] and discriminator loss function

[0028] The functional relationship is as follows: ;

[0029] ;

[0030] Where P represents the distribution of the true mask image. Here, Q represents the distribution of the occluded frames as input, where Q is the sampled data. For the sampling, Represents a generator. This represents the discriminator.

[0031] Compared with the prior art, this application has at least the following obvious advantages and effects:

[0032] This invention proposes a multi-dimensional cross-correlation enhancement technique to efficiently and selectively aggregate fine-grained contextual information from different dimensions based on their respective contributions, and to decompose the multi-dimensional cross-correlation enhancement into dimensions to reduce computational complexity. Through a pixel-level adversarial learning method, quality judgment is performed on the repaired video at the pixel level, mitigating the negative impact of large differences in data distribution while enhancing targeted optimization of the generator. The free-format video repair method integrates multi-dimensional cross-correlation enhancement and pixel-level adversarial learning, exhibiting originality and uniqueness in both algorithm and application. It fully considers the differences in the importance of different contextual information in the video and preserves temporal and spatial consistency in the reconstructed video, making the repaired video more realistic. Attached Figure Description

[0033] Figure 1 This is a structural diagram of the multidimensional cross-correlation video restoration model generator in this application.

[0034] Figure 2 This is a flowchart of the 3D gated convolution and the (2+1)D gated convolution combined with the residual mechanism in this application.

[0035] Figure 3 This is a schematic diagram of the two-dimensional cross-correlation calculation method in this application.

[0036] Figure 4 This is a schematic diagram of the cross-correlation enhancement in this application.

[0037] Figure 5 This is a flowchart of the multidimensional decomposition and cross-correlation calculation in this application.

[0038] Figure 6 This is a flowchart illustrating the structure of the discriminator in the pixel-level adversarial learning method of this application.

[0039] Figure 7 This is a structural diagram of the free-format video restoration network in this application.

[0040] Figure 8 This is a comparison chart of the latest algorithms for free-format video restoration in this application. Detailed Implementation

[0041] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention and not the entire structure.

[0042] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations (or steps) can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it may also have additional steps not included in the figures. The process may correspond to a method, function, procedure, subroutine, subroutine, etc.

[0043] The following embodiments and their optional solutions illustrate in detail the free-format video restoration method based on multi-dimensional information aggregation and pixel-level adversarial learning provided in this application.

[0044] See Figure 1 The Multidimensional Cross-Correlation (MDC) video inpainting model is used as a generator and consists of three parts: upsampling, dilation, and downsampling before and after. Downsampling is performed using (2+1)D gated convolutions with residual mechanisms ((2+1)D GatedConv), the MDC module adaptively and effectively aggregates contextual information, dilated convolutions expand the receptive field, and upsampling is performed using (2+1)D gated convolutions with residual mechanisms plus nearest neighbor interpolation (Deconv). Two MDC modules are inserted before and after the dilation module to collect information from the reference feature maps in the context.

[0045] See Figure 2 The gated convolution used in the multidimensional cross-correlation video restoration model is a special convolutional module used in image / video restoration. The left image shows a 3D gated convolution, and the rightmost image shows a (2+1)D gated convolution combined with a residual mechanism. In order to further reduce the model parameters, the original 3D convolution kernel shape is decomposed into 3D convolution and 3D convolution by combining the (2+1)D gated convolution with a residual mechanism, and this mechanism is applied in both types of convolution. In addition, the (2+1)D gated convolution and the residual mechanism are combined, and the (2+1)D gated convolution combined with the residual mechanism is used as the basic convolutional module of the multidimensional cross-correlation video restoration model, which significantly reduces the number of parameters of the multidimensional cross-correlation video restoration model (approximately 20M).

[0046] See Figure 3 Multidimensional cross-correlation can collect information in fine-grained terms from contextual information in different dimensions and adaptively assign corresponding weights to them according to different local information features; the steps of cross-correlation calculation in the simple 2-dimensional case are visualized to illustrate the more general multidimensional cross-correlation enhancement.

[0047] In a given 3D target feature map and a reference feature map that is in the same feature space and has the same shape , to a feature block Defined as an index A multidimensional feature tensor centered on the core; two feature blocks ( In target feature map Chinese-Israeli index Centered on, In the reference feature map Chinese-Israeli index The cross-correlation calculation (centered on) can be defined by the following formula: ;in yes The index is eigenvalues, yes The index is eigenvalues, It's a similarity calculation; it uses vector dot product as the similarity calculation method; the shape of the feature block is... , by each dimension The corresponding feature block radius Decide.

[0048] for Each point in Calculate one by one Target feature block centered and The cross-correlation of all reference feature blocks within the receptive domain, in In China, with The radius of the receptive field centered on the center is expressed as When extracting the reference feature block, we apply a step size in each dimension. Therefore, the index The set can be represented as ,in The final cross-correlation result graph. Defined as all in the target feature map and reference feature map The cross-correlation between feature blocks in the data, that is... ;

[0049] Will Operation application to First, obtain the cross-correlation weights. Then it will eventually be used to update Cross-correlation calculation results The following calculations were performed: ; The entire process described above can be simplified as follows: .

[0050] The cross-correlation weights are applied to the center values ​​of the corresponding reference feature blocks, and the final cross-correlation enhancement result is obtained by weighted summation.

[0051] See Figure 4 After cross-correlation enhancement of the reference and target images, residual components are extracted to enhance the cross-correlation calculation results. As shown in the visualization features, after cross-correlation calculation, more texture and structural details are aggregated from the reference features to the target features.

[0052] See Figure 5 In multidimensional cross-correlation video restoration models, feature maps have four dimensions: height, width, channels, and time. As the number of dimensions increases, the computational complexity of high-dimensional cross-correlation calculations also increases exponentially. To reduce computational complexity, dimensionality decomposition breaks down the four-dimensional cross-correlation calculation into a three-dimensional (space-time) cross-correlation calculation and a two-dimensional (time-channel) cross-correlation calculation. ;exist Figure 4 After cross-correlation enhancement, the texture and structure in the visualized feature map become clearer; the ST Correlation and T-CCorrelation modules in the map will both be used. Figure 4 The structure is as follows: two convolutional layers are used before and after the cross-correlation calculation to adjust the number of channels, thereby adjusting the memory usage. A gating mechanism is also added after the multidimensional cross-correlation calculation.

[0053] See Figure 6 In the pixel-level adversarial learning method (MaskGAN), the occlusion discriminator evaluates each pixel in the reconstructed video, distinguishing between the pixels generated by the generator and the original preserved pixels. The discriminator consists of a high-pass filter and a mask generator. By adding the high-pass filter, the high-frequency components in the restored video are highlighted, which can better help the discriminator identify restoration traces in the video. The mask generator is connected after the high-pass filter and is used to generate the final output of the pixel-level adversarial learning method discriminator, which is the predicted mask map.

[0054] The mask generator specifically consists of a downsampling module composed of gated convolutions and an upsampling module composed of pixel rearrangements. Pixel rearrangements transform a tensor of shape into a tensor of shape through dimension transformation and matrix deformation, thereby achieving upsampling. Using a high-pass filter and the mask generator, the pixel-level adversarial learning method can output a mask image that infers whether a part is real or fake. This mask image is used to distinguish between the repaired portion and the natural portion of the reconstructed frame. The generator loss function... and discriminator loss function It is expressed as follows:

[0055] ; ;

[0056] Where P represents the distribution of the true mask image. Here, Q represents the distribution of the occluded frames as input, where Q is the sampled data. For sampling. Represents a generator. This represents the discriminator.

[0057] See Figure 7 The Free Format Video Inpainting Network (MDCM) consists of a multidimensional cross-correlation video inpainting model as a generator and an occlusion discriminator in a pixel-level adversarial learning method. The occlusion discriminator evaluates the pixels in the reconstructed video and uses the multidimensional cross-correlation video inpainting model as a generator to distinguish between the generated pixels and the original preserved pixels.

[0058] See Figure 8 This image shows a comparison of state-of-the-art algorithms in free-format video inpainting with other algorithms. It compares a free-format video inpainting method integrating a multi-dimensional cross-correlation video inpainting model and pixel-level adversarial learning with other excellent video inpainting methods: Convolutional network variants: CombCN, 3D Gated, and LGTSM. CombCN is a two-stage deep video inpainting method using a 3D encoder-decoder network, 3D Gated is a single-stage deep video inpainting method using 3D gated convolutions and Temporal PatchGAN loss, LGTSM is a 2D convolutional method that aggregates contextual information using a temporal shift mechanism, and VINet is an optical flow-based method. VINet combines an image encoder-decoder structure and an optical flow model architecture. TCCDS is a well-known feature block matching-based video inpainting method. EdgeConnect (EC) is a two-stage generative adversarial learning network consisting of an edge generator and an image completion model.

[0059] It should be further noted that experiments were conducted on two common public datasets: FaceForensics and the free-format video restoration dataset FVI. This invention generates occlusions of different types (curves, objects, rectangles) and different occlusion-image ratios (ranging from 0%-10% to 60%-70%).

[0060] FaceForensics is a face dataset containing 1,004 video clips, where the faces are frontal and exhibit neutral cross-frame expression changes. These videos are from YouTube and the YouTube-8m dataset, tagged with faces, news anchors, or news programs. The videos have been cropped to... The face is in the center. Of these, 854 videos were used for training and 150 videos were used for testing.

[0061] The Free-Format Video Inpainting Dataset (FVI) offers high diversity, posing a greater challenge to video inpainting tasks. The FVI dataset contains various natural scenes, moving objects, animals, and human activities. It was trained using 1940 videos and tested with 100 videos. All videos in the dataset were resized and randomly cropped to a resolution of 320×180.

[0062] Table 1 compares the performance of different models on the FaceForensics dataset and under three types of occlusion. The bolded values ​​represent the best-performing models under these conditions. It can be seen that the proposed free-form inpainting method outperforms other models under various conditions and metrics.

[0063] Table 2 shows the performance of different models on the FVI dataset and under three types of occlusion. FVI is a more diverse and challenging dataset, and the model outperformed other models in almost all metrics. LPIPS demonstrates the semantic plausibility of the model's output video and its semantic similarity to the real video. FID indicates the temporal consistency of the output video and the overall distance between the restored video distribution and the real distribution. Compared to these two, MSE is a better indicator of the semantic and visual plausibility of the model's restored video. Figure 8 As shown, the model repair results presented are closer to the real-world scene.

[0064] Table 1. Experimental results of different methods on the FaceForensics dataset under the above three evaluation methods.

[0065]

[0066] Table 2 shows the experimental results of different methods on the FVI dataset under the three evaluation methods mentioned above.

[0067]

[0068] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A free-format video restoration method based on multi-dimensional information aggregation and pixel-level adversarial learning, characterized in that: The system includes a multidimensional cross-correlation video inpainting model used as a generator and an occlusion discriminator in a pixel-level adversarial learning method, wherein the multidimensional cross-correlation video inpainting model used as a generator and the occlusion discriminator in a pixel-level adversarial learning method are associated through a regenerated frame; The multidimensional cross-correlation video restoration model used as the generator consists of three parts: upsampling, dilation, and downsampling before and after. Multidimensional cross-correlation modules are inserted before and after the dilation module. The multidimensional cross-correlation module collects information from a given D-dimensional target feature map and a reference feature map in the same feature space, calculates the cross-correlation between the target feature map and the reference feature map, assigns corresponding cross-correlation weights based on the cross-correlation of the feature blocks, applies the cross-correlation weights to the center values ​​of the corresponding reference feature blocks, and obtains the final cross-correlation enhancement result by weighted summation. The computational complexity of cross-correlation between dimensions is reduced by reducing the dimensionality of the feature map. The occlusion discriminator in the pixel-level adversarial learning method consists of a high-pass filter and a mask generator connected one after the other. The high-pass filter and the mask generator output a mask image to infer whether it is real or fake. The occlusion discriminator evaluates the pixels in the reconstructed video and distinguishes the pixels generated by the mask generator from the original preserved pixels. The high-pass filter is used to highlight the high-frequency components in free-format video restoration, which helps the discriminator identify restoration traces in the video; The mask generator is connected after the high-pass filter and is used to generate the final output of the pixel-level adversarial learning discriminator for free-format video restoration; the mask image is used to distinguish the restored part and the natural part of the reconstructed frame. Mask generator loss function discriminator loss function The relationship is: ； ; Where P represents the distribution of the true mask image, x represents the sampled portion of it, Q represents the distribution of the occluded frame as input, z represents the sampled portion of it, G represents the generator, and D represents the discriminator.

2. The free-form video inpainting method based on multi-dimensional information aggregation and pixel-level adversarial learning according to claim 1, characterized in that, When the multidimensional cross-correlation video restoration model is used as a generator, downsampling is achieved by combining (2+1)D gated convolution with residual mechanism, and upsampling is achieved by combining (2+1)D gated convolution with residual mechanism plus nearest neighbor interpolation.

3. The free-form video inpainting method based on multi-dimensional information aggregation and pixel-level adversarial learning according to claim 2, characterized in that, The (2+1)D gated convolution with residual mechanism combines (2+1)D gated convolution with residual mechanism. The (2+1)D gated convolution with residual mechanism serves as the basic convolutional module of the multidimensional cross-correlation video restoration model, which is used to reduce the number of parameters in the multidimensional cross-correlation video restoration model.

4. The free-form video inpainting method based on multi-dimensional information aggregation and pixel-level adversarial learning according to claim 1, characterized in that, The expansion module adaptively and effectively aggregates the contextual information obtained from upsampling and downsampling in the reference feature map by employing a multidimensional cross-correlation module, and expands the receptive field using dilated convolution.

5. The free-form video inpainting method based on multi-dimensional information aggregation and pixel-level adversarial learning according to claim 1, characterized in that, The target feature block With target feature map Chinese-Israeli index Centered on, reference feature block With reference feature map Chinese-Israeli index Centered on this, we obtain the formula for calculating cross-correlation: ; and Given a D-dimensional target feature map and a reference feature map, where yes The index is eigenvalues, yes The index is eigenvalues, It's a similarity calculation; the shape of the feature block is... , by each dimension The corresponding feature block radius Decide.

6. The free-format video restoration method based on multi-dimensional information aggregation and pixel-level adversarial learning according to claim 1, characterized in that, The D-dimensional target feature map has four dimensions: height, width, channel, and time. Due to the high dimensionality, the computation is complex. Dimensionality reduction is used to address this computational difficulty, and the calculation is as follows: ; Dimensional decomposition breaks down the 4-dimensional cross-correlation calculation into a 3-dimensional cross-correlation calculation (height-width-time) and a 2-dimensional cross-correlation calculation (time-channel).

Images