An end-to-end video inpainting method and system based on optical flow guidance

Abstract

This invention belongs to the field of computer vision technology and provides an end-to-end video completion method and system based on optical flow guidance. The method includes: acquiring a video to be processed; segmenting the video to be processed into local neighborhood video frames and non-local reference video frames; converting the local neighborhood video frames and non-local reference video frames into local neighborhood features and non-local reference features using an encoder; performing end-to-end inter-frame optical flow estimation and completion based on the local neighborhood video frames using an optical flow completion module to obtain completed bidirectional optical flow; using the local neighborhood features and the completed bidirectional optical flow, performing local information supplementation based on optical flow guidance using a feature propagation module to obtain supplemented local neighborhood features; and obtaining the completed video based on the non-local reference features and the supplemented local neighborhood features.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to an end-to-end video completion method and system based on optical flow guidance. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] Video completion aims to fill in the "damaged" regions with plausible and coherent content in video clips. It is widely used in real-world applications such as object removal, video inpainting and video extension. Although significant progress has been made in image completion, video inpainting remains challenging due to complex video scenes and excessively damaged video frames. If video completion is performed independently on each frame directly, it often generates temporally inconsistent videos and leads to severe artifacts. Therefore, high-quality video completion needs to consider both spatial structure and temporal correlation. The latest progress in deep learning has prompted researchers to develop more effective solutions.

[0004] Based on deep learning technology, video completion has made great progress. These video completion methods can be roughly divided into three categories: 3D convolution-based methods, optical flow-guided methods, and attention-based methods. Some methods using 3D convolution and attention often produce temporally inconsistent results due to the limited receptive field in the temporal dimension. In order to produce more temporally consistent results, many works regard optical flow as a strong prior for video completion and try to directly embed optical flow estimation into the network. However, it is very difficult to directly calculate the inter-frame optical flow in the invalid region because these regions themselves become occlusion factors, thus limiting performance. Therefore, the latest optical flow-based methods consider to complete the missing optical flow estimates. These methods can be decomposed into three interrelated stages. (1) Optical flow completion: Since the missing motion information in the damaged region cannot be known, it is necessary to first complete the pre-estimated optical flow; (2) Pixel propagation: Under the guidance of the completed optical flow, pixel values ​​are propagated bidirectionally in the visible region to fill the missing parts in the damaged video; (3) Content reconstruction: After propagation, the remaining missing regions in the video can be filled by a pre-trained image completion network. However, while these methods can achieve good visual results, the entire optical flow-based video completion process must be performed separately because the first two stages involve many manually defined operations (e.g., Poisson mixing, solving sparse equations, and indexing optical flow trajectories by pixel). These independently deployed processes raise two main problems: First, errors in the early stages accumulate and amplify in later stages, significantly affecting the final completion result. Specifically, inaccurate optical flow estimation can mislead pixel propagation and further obfuscate content reconstruction, leading to unreliable video completion results. Second, these complex, manually defined operations can only be performed without GPU acceleration. Therefore, the entire inference process on the video sequence is very time-consuming. For example, completing a 432×240 video with 70 frames per second from the DFVI dataset takes approximately 4 minutes, which is unacceptable in most practical applications. Furthermore, in addition to the above drawbacks, using only a pre-trained image completion network in the content reconstruction stage ignores content continuity across temporal neighborhoods, leading to inconsistencies in the generated content within the video. In addition, some recent attention-based video completion frameworks have achieved good results, but because they do not take into account the motion relationships between frames well, they produce some unreliable generation results. Summary of the Invention

[0005] To address the issues of low computational efficiency, suboptimal generated results, and lack of spatiotemporal consistency in existing video completion frameworks, this invention provides an end-to-end video completion method and system based on optical flow guidance. This method generates optical flow in an end-to-end manner and replaces the pixel-based propagation method in previous frameworks with a feature-level propagation approach. This invention enables efficient video completion training and yields accurate video completion results with strong temporal consistency.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] The first aspect of the present invention provides an end-to-end video completion method based on optical flow guidance.

[0008] An end-to-end video completion method based on optical flow guidance includes:

[0009] Get the video to be processed;

[0010] The video to be processed is divided into local neighborhood video frames and non-local reference video frames.

[0011] The encoder transforms local neighborhood video frames and non-local reference video frames into local neighborhood features and non-local reference features.

[0012] Based on the local neighborhood video frames, an optical flow completion module is used to perform end-to-end inter-frame optical flow estimation and completion to obtain the completed bidirectional optical flow.

[0013] Based on local neighborhood features and the completed bidirectional optical flow, a feature propagation module is used to supplement local information guided by optical flow, resulting in supplemented local neighborhood features.

[0014] Based on the non-local reference features and the supplemented local neighborhood features, the completed video is obtained.

[0015] A second aspect of the present invention provides an end-to-end video completion system based on optical flow guidance.

[0016] An end-to-end video completion system based on optical flow guidance, comprising:

[0017] The data acquisition module is configured to acquire the video to be processed.

[0018] The preprocessing module is configured to divide the video to be processed into local neighborhood video frames and non-local reference video frames.

[0019] The encoder module is configured to convert local neighborhood video frames and non-local reference video frames into local neighborhood features and non-local reference features through an encoder.

[0020] The optical flow completion module is configured to perform end-to-end inter-frame optical flow estimation and completion on the local neighborhood video frames to obtain the completed bidirectional optical flow.

[0021] The feature propagation module is configured to supplement local neighborhood features and the completed bidirectional optical flow with local information based on optical flow guidance to obtain supplemented local neighborhood features.

[0022] The output module is configured to obtain the completed video based on the non-local reference features and the supplemented local neighborhood features.

[0023] A third aspect of the present invention provides a computer-readable storage medium.

[0024] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the optical flow-guided end-to-end video completion method as described in the first aspect above.

[0025] A fourth aspect of the present invention provides a computer device.

[0026] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the optical flow-guided end-to-end video completion method as described in the first aspect above.

[0027] Compared with the prior art, the beneficial effects of the present invention are:

[0028] This invention implements an optical flow-guided end-to-end video completion framework by designing three end-to-end trainable modules. This end-to-end framework makes the training and deployment process more efficient, while also reducing error accumulation and amplification, thus contributing to more accurate video completion results. By considering the relationship between local and non-local neighborhoods simultaneously during content reconstruction, the framework can generate more temporally consistent video completion results. Attached Figure Description

[0029] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0030] Figure 1 This is a schematic diagram of the end-to-end video completion framework based on optical flow guidance in this invention;

[0031] Figure 2 This is a schematic diagram of the feature propagation module based on deformable convolution in this invention;

[0032] Figure 3This is a schematic diagram illustrating the effect of applying the present invention to a video restoration task;

[0033] Figure 4 This is a schematic diagram illustrating the effect of applying the present invention to a target removal task. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0035] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0036] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0037] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems according to various embodiments of this disclosure. It should be noted that each block in a flowchart or block diagram may represent a module, segment, or portion of code, which may include one or more executable instructions for implementing the logical functions specified in the various embodiments. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutively represented blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, may be implemented using a dedicated hardware-based system that performs the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0038] Example 1

[0039] This embodiment provides an end-to-end video completion method based on optical flow guidance. This embodiment uses the application of this method to a server as an example for illustration. It is understood that this method can also be applied to terminals, and can also be applied to systems including terminals, servers, and other components, and can be implemented through interaction between the terminal and the server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communication, middleware services, domain name services, CDN security services, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein. In this embodiment, the method includes the following steps:

[0040] Get the video to be processed;

[0041] The video to be processed is divided into local neighborhood video frames and non-local reference video frames.

[0042] The encoder transforms local neighborhood video frames and non-local reference video frames into local neighborhood features and non-local reference features.

[0043] Based on the local neighborhood video frames, an optical flow completion module is used to perform end-to-end inter-frame optical flow estimation and completion to obtain the completed bidirectional optical flow.

[0044] Based on local neighborhood features and the completed bidirectional optical flow, a feature propagation module is used to supplement local information guided by optical flow, resulting in supplemented local neighborhood features.

[0045] Based on the non-local reference features and the supplemented local neighborhood features, the completed video is obtained.

[0046] Reference Figure 1 The end-to-end video completion framework based on optical flow guidance is as follows:

[0047] a. The input video frames are divided into local neighborhood video frames and non-local reference video frames according to their neighborhood relationship. The local neighborhood video frames and non-local reference video frames are fed into the encoder together to obtain local neighborhood features and non-local reference features.

[0048] b. Local neighborhood video frames are fed into the optical flow completion module for end-to-end inter-frame optical flow estimation and completion. During training, both the forward and backward optical flows estimated by SPyNet can be constrained by the L1 loss function to achieve optical flow completion. Assume the predicted forward optical flow and the actual forward optical flow are respectively... and F t→t+1 The predicted reverse optical flow and the actual reverse optical flow are respectively and F t→t-1 Then we have:

[0049]

[0050] c. The encoded local neighborhood feature domain, along with the completed bidirectional optical flow, is fed into a feature propagation module based on deformable convolution to complete the content information that can be traced by the optical flow. Taking the forward optical flow being fed into this module as an example, ... Figure 2 As shown, the weight mask W needs to be calculated first. t→t+1 And the offset ΔF relative to the optical flow t→t+1 The calculation method can be expressed as:

[0051]

[0052] Among them, E t The features of the current time step t obtained by the encoder; This represents multiple stacked convolutional layers; The purpose of this is to transform the deformation function. according to The deformation is applied to the current time step. Next, the module uses deformable convolutions to align the inverse features. To obtain the updated inverse features This process can be represented as:

[0053]

[0054] in, This represents a deformable convolution operation; The backpropagation function can be composed of one or more convolutional layers or residual modules. The weight mask W... t→t+1 The effectiveness of each point is measured by normalizing it using the sigmoid function and then applying it to each sampling point. Applying the above operation in the reverse direction to the forward features yields... Finally, the forward and backward features can be fused together using a 1×1 convolutional layer to output the final propagated features.

[0055] Among them, the feature propagation module is a bidirectional recurrent neural network structure, and the completed local neighborhood optical flow can be used as the initial bias estimate for deformable convolution.

[0056] d. The propagated local features can be fused with non-local reference features and fed into the content reconstruction module to complete the remaining content. To ensure a certain degree of temporal consistency, this content reconstruction module considers not only local neighborhood relationships in the temporal dimension but also non-local neighborhood relationships. In terms of the spatial dimension, the content reconstruction module performs coarse-grained attention calculations on global information and fine-grained attention calculations on local information. After the calculations are completed, the completed local features and non-local reference features are obtained. The completed local features and non-local reference features are then fed into the decoder to output the completed video frame.

[0057] It should be noted that the network structure used for optical flow completion can be any network structure used for optical flow estimation; its initialization parameters can be set to the network parameters used for optical flow estimation.

[0058] The optical flow-guided end-to-end video completion framework provided in this embodiment can perform efficient video completion training and obtain accurate video completion results with strong temporal consistency. This framework can be applied to video editing tasks including, but not limited to, video restoration and recovery, video watermark removal, video subtitle removal, and video object removal.

[0059] To demonstrate the effectiveness of this embodiment's design, it was compared with state-of-the-art frameworks on the YouTube VOS dataset. These state-of-the-art frameworks include DFVI, FGVC, STTN, and FuseFormer. The metrics used for comparison included Peak Signal-to-Noise Ratio (PSNR), Structure Similarity (SSIM), Video Frechet Inception Distance (VFID), and Temporal Warping Error. The results are shown in Table 1, demonstrating that this framework outperforms other frameworks in all metrics.

[0060] Table 1

[0061] Method PSNR↑ SSIM↑ VFID↓ Ewarp↓ DFVI 29.16 0.9429 0.066 0.1509 FGVC 29.67 0.9403 0.064 0.1022 STTN 32.34 0.9655 0.053 0.0907 FuseFormer 33.29 0.9681 0.053 0.0900 The framework 33.71 0.9700 0.046 0.0864

[0062] To further illustrate the effectiveness of this embodiment, the framework was compared with other frameworks in terms of efficiency. The comparison metrics included floating-point operations per second (FLOPs) and processing time per frame. FLOPs were calculated using an 8-frame neighborhood, and processing time per frame was calculated using a single Titan XP GPU on the YouTube VOS dataset. The results are shown in Table 2. This framework has a similar computation time to attention-based video completion frameworks (including STTN and FuseFormer), and is nearly 15 times faster than optical flow-based frameworks (including DFVI and FGVC). Furthermore, this framework has a lower computational cost compared to attention-based video completion frameworks.

[0063] Table 2

[0064]

[0065]

[0066] Reference Figure 3 In video restoration tasks, the framework in this embodiment can more accurately restore the details and structural information of objects compared to other video completion frameworks, and generate results with a certain degree of temporal consistency. Other frameworks, on the other hand, generate video frames that contain distorted details and unreasonable content.

[0067] Reference Figure 4 In the target removal task, the framework in this embodiment can more accurately restore the details of the previously occluded parts of the object after removal compared to other video completion frameworks. Other frameworks, on the other hand, generate video frames that fail to accurately restore the object's details and exhibit poor spatiotemporal consistency.

[0068] Example 2

[0069] This embodiment provides an end-to-end video completion system based on optical flow guidance.

[0070] An end-to-end video completion system based on optical flow guidance, comprising:

[0071] The data acquisition module is configured to acquire the video to be processed.

[0072] The preprocessing module is configured to divide the video to be processed into local neighborhood video frames and non-local reference video frames.

[0073] The encoder module is configured to convert local neighborhood video frames and non-local reference video frames into local neighborhood features and non-local reference features through an encoder.

[0074] The optical flow completion module is configured to perform end-to-end inter-frame optical flow estimation and completion on the local neighborhood video frames to obtain the completed bidirectional optical flow.

[0075] The feature propagation module is configured to supplement local neighborhood features and the completed bidirectional optical flow with local information based on optical flow guidance to obtain supplemented local neighborhood features.

[0076] The output module is configured to obtain the completed video based on the non-local reference features and the supplemented local neighborhood features.

[0077] It should be noted that the data acquisition module, preprocessing module, encoder module, optical flow completion module, feature propagation module, and output module described above are the same examples and application scenarios implemented in Embodiment 1, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0078] Example 3

[0079] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the optical flow-guided end-to-end video completion method as described in Embodiment 1 above.

[0080] Example 4

[0081] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the optical flow-guided end-to-end video completion method as described in Embodiment 1 above.

[0082] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0083] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0084] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0085] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0086] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0087] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An end-to-end video inpainting method based on optical flow guidance, characterized in that, The method comprises the following steps: acquiring a video to be processed; segmenting the video to be processed into local neighborhood video frames and non-local reference video frames; translating the local neighborhood video frames and the non-local reference video frames into local neighborhood features and non-local reference features through an encoder; based on the local neighborhood video frames, performing end-to-end inter-frame optical flow estimation and completion on the local neighborhood video frames through an optical flow completion module to obtain completed bidirectional optical flow; based on the local neighborhood features and the completed bidirectional optical flow, performing optical flow guided local information supplement on the local neighborhood features and the completed bidirectional optical flow through a feature propagation module to obtain supplemented local neighborhood features; based on the non-local reference features and the supplemented local neighborhood features, obtaining a completed video.

2. The optical flow guided end-to-end video completion method according to claim 1, wherein, based on the non-local reference features and the supplemented local neighborhood features, performing content reconstruction on the non-local reference features and the supplemented local neighborhood features through a content reconstruction module to obtain reconstructed video features.

3. The optical flow guided end-to-end video completion method according to claim 2, wherein, based on the reconstructed video features, performing decoding on the reconstructed video features through a decoder to obtain the completed video.

4. The optical flow guided end-to-end video completion method according to claim 1, wherein, The method further comprises the following steps: in a training stage, based on the local neighborhood video frames, estimating forward optical flow and backward optical flow through a SpyNet optical flow network; optimizing the optical flow network through a loss function to obtain an optimized optical flow network; obtaining the completed bidirectional optical flow through the optimized optical flow network.

5. The optical flow guided end-to-end video completion method according to claim 4, wherein, The completed bidirectional optical flow comprises completed forward optical flow and completed backward optical flow.

6. The optical flow guided end-to-end video completion method according to claim 5, wherein, calculating a weight mask of the completed forward optical flow and an offset relative to the forward optical flow, aligning reverse features through deformable convolution of the feature propagation module according to the weight mask of the completed forward optical flow and the offset relative to the forward optical flow, and then combining a convolution layer or a residual module to obtain updated reverse features; calculating a weight mask of the completed backward optical flow and an offset relative to the backward optical flow, aligning forward features through deformable convolution of the feature propagation module according to the weight mask of the completed backward optical flow and the offset relative to the backward optical flow, and then combining a convolution layer or a residual module to obtain updated forward features; fusing the updated reverse features and the forward features to obtain propagated features.

7. An end-to-end video inpainting system based on optical flow guidance, characterized in that, The method comprises the following steps: a data acquisition module configured to acquire a video to be processed; a preprocessing module configured to segment the video to be processed into local neighborhood video frames and non-local reference video frames; an encoder module configured to translate the local neighborhood video frames and the non-local reference video frames into local neighborhood features and non-local reference features through an encoder; an optical flow completion module configured to perform end-to-end inter-frame optical flow estimation and completion on the local neighborhood video frames to obtain completed bidirectional optical flow; a feature propagation module configured to perform optical flow guided local information supplement on the local neighborhood features and the completed bidirectional optical flow to obtain supplemented local neighborhood features; an output module configured to obtain a completed video based on the non-local reference features and the supplemented local neighborhood features.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by a processor to implement the steps in the optical flow guided end-to-end video completion method according to any one of claims 1-6.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps in the light flow guided end-to-end video completion method as claimed in any one of claims 1-6 when executing the program.

Images