Video restoration method and system guided by deep prior optical flow

By combining classic semi-quadratic splitting algorithms and deep learning, using optical flow to guide local attention for video repair, the problems of complexity and computational overhead of video repair methods in the existing technology are solved, and efficient and interpretable video repair is achieved. Effect.

CN117994176BActive Publication Date: 2024-11-15COMMUNICATION UNIVERSITY OF CHINA +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311823811.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-27
Publication Date
2024-11-15
Estimated Expiration
2043-12-27

AI Technical Summary

Technical Problem

The existing video repair methods have problems such as complex network design, large computing overhead, and difficult to guarantee space-time consistency, especially in deep learning methods, the lack of interpretability and insufficient utilization of relevant knowledge in the video field.

Method used

The classic semi-quadratic splitting algorithm is used in combination with deep learning, and the unconstrained optimization problem is converted into a constraint optimization problem by introducing auxiliary variables, and a deep prior video framework network with fixed number of stages is designed, and local attention is used to guide local attention for video repair. Combining traditional methods and deep learning to improve network performance.

Benefits of technology

It reduces network design structure and computing overhead, improves video repair efficiency, enhances the time and space consistency and interpretability of the results, and performs better than existing advanced video repair methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117994176B_ABST
    Figure CN117994176B_ABST
Patent Text Reader

Abstract

The present invention provides a deep a priori optical flow guided video restoration method. First, a target video is input into a deep a priori video framework network with a fixed number of stages to iterate the target video in stages. Each stage iteration corresponds to a classic semi-quadratic splitting algorithm, and in each iteration process, a preset data module extracts and outputs data for the target video, a preset reconstruction module locks the video to be restored in the target video, and performs video restoration based on optical flow guided local attention on the video to be restored to obtain a restored video. In this way, a traditional model-driven method and a data-driven method based on deep learning are combined to form a deep a priori video framework with a fixed number of stages. The traditional method and the method based on deep learning are combined to improve network performance, improve interpretability and reliability, reduce network design structure, reduce computational overhead, and improve video restoration efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer video processing technology, and more specifically, to a method and system for video restoration guided by deep a priori optical flow. Background Art

[0002] Video inpainting refers to the process of completing missing areas in a video using information from known areas in the entire video sequence. It is widely used in practical applications such as video editing and video supervision. As an extension of image inpainting, although research in the former field has made significant progress, video inpainting still faces severe challenges from multiple aspects due to the introduction of an additional time dimension. If the video is split frame by frame and the existing image inpainting methods are directly applied for inpainting, it may lead to temporal inconsistencies and produce obvious flickering artifacts. Therefore, how to aggregate the spatiotemporal information within and between frames to ensure the spatiotemporal consistency of the inpainted results is very important for the video inpainting problem. With the rise of deep learning, more and more researchers have begun to turn their attention to the research of video inpainting methods based on deep learning.

[0003] In recent years, many deep learning-based video restoration methods have emerged. These methods can be roughly divided into three directions: methods based on three-dimensional convolutional networks, methods based on optical flow, and methods based on attention mechanisms. The methods based on three-dimensional convolutional networks use their learning adaptability to automatically capture changes in time and space, and implicitly align and fuse features. This method has a simple structure and is usually trained in an end-to-end manner. However, the spatiotemporal receptive field of the three-dimensional convolutional network is limited, and it is difficult to fuse the effective information of video frames that are far away. Although increasing the convolution kernel can expand the receptive field, it will also increase the number of model parameters. The optical flow-based method uses optical flow to capture the dynamic motion of the video, and uses a two-dimensional convolutional network to calculate the three-dimensional pattern, saving computational overhead. However, the video restoration method based on optical flow is highly dependent on the results of optical flow estimation. Once the optical flow estimation results are wrong, it will affect the subsequent feature propagation and alignment. The method based on the attention mechanism can effectively capture long-range dependencies, but because the search range of the attention mechanism used in video restoration is wider, the amount of computation will also be relatively large. Similar to the image restoration method based on deep learning, the video restoration method based on deep learning has better performance, but it also has problems such as lack of interpretability due to the "black box" operation of deep learning and failure to fully utilize relevant knowledge in the field of video restoration. Moreover, due to the high dimensionality of the video, the above problems will further lead to more complex network design and difficulty in training and converging the network.

[0004] Therefore, there is an urgent need for a deep prior optical flow guided video restoration method and system that can reduce network design structure, reduce computational overhead, and improve video restoration efficiency. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to provide a deep prior optical flow guided video restoration method and system to solve the problems that existing image processing methods either process too much data, do not combine financial image attributes, or have poor privacy.

[0006] The present invention provides a deep priori optical flow guided video restoration, wherein:

[0007] When the classic semi-quadratic splitting algorithm is used to optimize the target video, the semi-quadratic splitting algorithm is used to solve the optimization problem. By introducing an auxiliary variable Z, the formula The unconstrained optimization problem described is transformed into a constrained optimization problem:

[0008] in, Represents the original undamaged video, is a frame-by-frame binary mask indicating the missing area in the video, ⊙ represents the point-by-point multiplication of the corresponding elements, The damaged target video is obtained by multiplying the original video and the mask frame by frame. represents the restored video; and the formula It can be reformulated as the following optimization problem:

[0009]

[0010] Among them, μ is a penalty parameter;

[0011] The semi-quadratic splitting algorithm iteratively solves between the following two equations:

[0012]

[0013] formula It is related to the quadratic regularized least squares problem, whose closed form solution is as follows:

[0014]

[0015] where α k-1 =μ k-1 σ 2 ,formula The prior term Φ(X) in Φ(X) is related to the original uncorrupted video and can only be estimated from the corrupted video in practice.

[0016] The formula and The update steps of the semi-quadratic splitting algorithm are mapped into a network architecture consisting of multiple stages, each of which corresponds to an iteration in the semi-quadratic splitting algorithm. Each stage is solved by the formula It consists of a data module and a reconstruction module for learning deep priors;

[0017] Where k represents the iteration index of the semi-quadratic splitting algorithm; μ is set to the maximum, μ is increased in an iterative manner, and μ used in the kth iteration is replaced by μ k express;

[0018] The target video is input into a deep a priori video framework network with a fixed number of stages to iterate the target video in stages, each stage iteration corresponds to a classic semi-quadratic splitting algorithm, and in each iteration process, a preset data module extracts and outputs data from the target video, a preset reconstruction module locks the video to be repaired in the target video, and performs video repair based on optical flow guiding local attention on the video to obtain a repaired video; wherein the step of obtaining a repaired video by performing video repair based on optical flow guiding local attention includes:

[0019] The video to be repaired is divided into local continuous adjacent frames and global reference frames, and the local continuous adjacent frames are converted into local continuous adjacent features and the global reference frames are converted into global reference features through a preset encoder;

[0020] Using a preset optical flow completion module to predict and complete the local continuous adjacent frames between each other to obtain a bidirectional optical flow and a bidirectional optical flow occlusion mask, and obtaining a supplemented aligned local continuous adjacent feature based on the bidirectional optical flow and the local continuous adjacent feature;

[0021] Using a preset inter-frame restoration module to obtain a global time information video feature according to the local continuous adjacent features after the complementary alignment and the global reference features;

[0022] The global time information video feature is decoded by a preset decoder to obtain a restored video.

[0023] Preferably, the optical flow completion module is a pre-trained module; wherein the step of training the optical flow completion module comprises:

[0024] Inputting the sample loss video into the optical flow estimation network SpyNet, so that the optical flow estimation network SpyNet obtains and completes the forward optical flow and the backward optical flow of the sample loss video, and learns the sample occlusion mask with pixel visibility in the forward optical flow and the backward optical flow;

[0025] The optical flow estimation network SpyNet is optimized based on the sample occlusion mask and the preset occlusion mask of the complete video sample using a preset loss function to obtain an optical flow network until the data of the loss function reaches a preset threshold, and the optical flow network trained for the last time is used as an optical flow completion module.

[0026] Preferably, the bidirectional optical flow includes a completed forward optical flow and a completed backward optical flow.

[0027] Preferably, the step of acquiring the supplementally aligned local continuous adjacent features based on the bidirectional optical flow and the local continuous adjacent features comprises:

[0028] Selecting an undamaged frame from the local continuous adjacent frames as a reference frame;

[0029] The reference frame is wrapped around the current frame to be repaired by optical flow convolution for rough alignment, and the damaged optical flow gap of the current frame to be repaired is locked based on the local continuous adjacent features;

[0030] searching the bidirectional optical flow for an occlusion mask with pixel visibility corresponding to the damaged optical flow gap;

[0031] The occlusion mask with strong local attention is replaced on the local continuous neighboring features corresponding to the damaged optical flow gap to obtain the local continuous neighboring features after complementary alignment.

[0032] Preferably, the step of acquiring the global time information video feature according to the local continuous adjacent features after the complementary alignment and the global reference feature comprises:

[0033] Performing time axis alignment on the local continuous adjacent features after the complementary alignment and the global reference features;

[0034] In the same time period, comparing the local continuous adjacent features after the supplementary alignment with the global reference features to obtain video status parameters;

[0035] If the video status parameter indicates that the local continuous adjacent features after the supplementary alignment are better than the global reference features, then in the same time period, the global reference features are replaced by the local continuous adjacent features after the supplementary alignment to form the global time information video features.

[0036] The present invention also provides a deep a priori optical flow guided video restoration system, which implements the deep a priori optical flow guided video restoration method as described above, including a deep a priori video framework network with a fixed number of stages, which is used to receive a target video and iterate the target video in stages, each stage iteration corresponding to a classic semi-quadratic splitting algorithm;

[0037] The deep a priori video framework network includes a data module and a reconstruction module; the data module is used to extract and output data from the target video in each iteration process; the reconstruction module is used to lock the video to be repaired in the target video, and perform video repair based on optical flow to guide local attention on the video to be repaired to obtain a repaired video; wherein,

[0038] When the classic semi-quadratic splitting algorithm is used to optimize the target video, the semi-quadratic splitting algorithm is used to solve the optimization problem. By introducing an auxiliary variable Z, the formula The unconstrained optimization problem described is transformed into a constrained optimization problem:

[0039] in, Represents the original undamaged video, is a frame-by-frame binary mask indicating the missing area in the video, ⊙ represents the point-by-point multiplication of the corresponding elements, The damaged target video is obtained by multiplying the original video and the mask frame by frame. represents the restored video; and the formula It can be reformulated as the following optimization problem:

[0040]

[0041] Among them, μ is a penalty parameter;

[0042] The semi-quadratic splitting algorithm iteratively solves between the following two equations:

[0043]

[0044] Where k represents the iteration index of the semi-quadratic splitting algorithm; μ is set to the maximum, μ is increased in an iterative manner, and μ used in the kth iteration is replaced by μ k express;

[0045] formula It is related to the quadratic regularized least squares problem, whose closed form solution is as follows:

[0046]

[0047] where α k-1 =μ k-1 σ 2 ,formula The prior term Φ(X) in Φ(X) is related to the original uncorrupted video and can only be estimated from the corrupted video in practice.

[0048] The formula and The update steps of the semi-quadratic splitting algorithm are mapped into a network architecture consisting of multiple stages, each of which corresponds to an iteration in the semi-quadratic splitting algorithm. Each stage is solved by the formula It consists of a data module and a reconstruction module for learning deep priors;

[0049] The reconstruction module comprises:

[0050] A video data processing module, used for dividing the video to be repaired into local continuous adjacent frames and global reference frames;

[0051] An encoder, configured to convert the local continuous adjacent frames into local continuous adjacent features, and convert the global reference frame into a global reference feature;

[0052] The optical flow completion module is used to predict and complete the local continuous adjacent frames between each other to obtain the bidirectional optical flow and bidirectional optical flow occlusion mask;

[0053] An optical flow-guided feature propagation and alignment module, used for acquiring supplementary aligned local continuous adjacent features based on the bidirectional optical flow and the local continuous adjacent features;

[0054] An inter-frame restoration module, used for acquiring a global time information video feature according to the local continuous adjacent features after the complementary alignment and the global reference features;

[0055] A decoder is used to decode the global time information video feature to obtain a restored video.

[0056] Preferably, the optical flow completion module is a pre-trained module; wherein the step of training the optical flow completion module comprises:

[0057] Inputting the sample loss video into the optical flow estimation network SpyNet, so that the optical flow estimation network SpyNet obtains and completes the forward optical flow and the backward optical flow of the sample loss video, and learns the sample occlusion mask with pixel visibility in the forward optical flow and the backward optical flow;

[0058] The optical flow estimation network SpyNet is optimized based on the sample occlusion mask and the preset occlusion mask of the complete video sample using a preset loss function to obtain an optical flow network until the data of the loss function reaches a preset threshold, and the optical flow network trained for the last time is used as an optical flow completion module.

[0059] Preferably, the optical flow-guided feature propagation and alignment module includes a guided alignment module and a local multi-head attention mechanism.

[0060] Preferably, the guided alignment module is used to select an undamaged frame from the local continuous adjacent frames as a reference frame; wrap the reference frame around the current frame to be repaired for rough alignment through optical flow convolution, and lock the damaged optical flow gap of the current frame to be repaired based on the local continuous adjacent features; search for an occlusion mask with pixel visibility corresponding to the damaged optical flow gap in the bidirectional optical flow;

[0061] The local multi-head attention mechanism is used to obtain the local attention of the occlusion mask, and replace the occlusion mask with strong local attention on the local continuous adjacent features corresponding to the damaged optical flow gap to obtain the local continuous adjacent features after supplementary alignment.

[0062] Preferably, the step of the inter-frame repair module acquiring the global time information video feature according to the local continuous adjacent features after the supplementary alignment and the global reference features comprises:

[0063] The step of acquiring the global time information video feature according to the local continuous adjacent features after the complementary alignment and the global reference feature comprises:

[0064] Performing time axis alignment on the local continuous adjacent features after the complementary alignment and the global reference features;

[0065] In the same time period, comparing the local continuous adjacent features after the supplementary alignment with the global reference features to obtain video status parameters;

[0066] If the video status parameter indicates that the local continuous adjacent features after the supplementary alignment are better than the global reference features, then in the same time period, the global reference features are replaced by the local continuous adjacent features after the supplementary alignment to form the global time information video features.

[0067] It can be seen from the above technical scheme that the deep prior optical flow guided video restoration method and system provided by the present invention first inputs the target video into a deep prior video framework network with a fixed number of stages to iterate the target video in stages, and each stage iteration corresponds to a classic semi-quadratic splitting algorithm, and in each iteration process, a preset data module extracts and outputs data from the target video, a preset reconstruction module locks the video to be restored in the target video, and performs video restoration based on optical flow guided local attention on the video to be restored to obtain a restored video; wherein, the video to be restored is first divided into local continuous adjacent frames and global reference frames, and the local continuous adjacent frames are converted into local continuous adjacent features through a preset encoder, and the global reference frames are converted into global reference features, and then a preset The optical flow completion module predicts and completes the local continuous adjacent frames between each other to obtain bidirectional optical flow and bidirectional optical flow occlusion mask, and obtains the local continuous adjacent features after supplementary alignment based on the bidirectional optical flow and local continuous adjacent features. Then, a preset inter-frame repair module is used to obtain the global time information video features according to the local continuous adjacent features after supplementary alignment and the global reference features. Finally, the global time information video features are decoded by a preset decoder to obtain the repaired video. In this way, the traditional model-driven method and the data-driven method based on deep learning are combined to form a deep prior video framework with a fixed number of stages. The combination of traditional methods and deep learning-based methods improves network performance, improves interpretability and reliability, reduces network design structure, reduces computational overhead, and improves video repair efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] By referring to the following description in conjunction with the accompanying drawings, and with a more comprehensive understanding of the present invention, other objects and results of the present invention will become more clear and easy to understand. In the accompanying drawings:

[0069] Figure 1 is a flowchart of a method for video restoration guided by a depth prior optical flow according to an embodiment of the present invention;

[0070] Figure 2 A schematic diagram of a deep prior video framework network with a fixed number of stages in a deep prior optical flow guided video restoration method according to an embodiment of the present invention;

[0071] Figure 3 It is a schematic diagram of the process of a reconstruction module in a video restoration method guided by a depth prior optical flow according to an embodiment of the present invention;

[0072] Figure 4 4 is a system framework diagram of a deep prior optical flow guided video restoration system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0073] There are three video restoration methods in the prior art, including methods based on three-dimensional convolutional networks, methods based on optical flow, and methods based on attention mechanisms. The three-dimensional convolutional network-based method uses its learning adaptability to automatically capture changes in time and space. This method has a limited spatiotemporal receptive field and is difficult to fuse effective information from video frames that are far away. Although increasing the convolution kernel can expand the receptive field, it will also increase the number of model parameters. The optical flow-based method uses optical flow to capture the dynamic motion of the video and relies on the results of optical flow estimation. Once the optical flow estimation results are wrong, it will affect the subsequent feature propagation and alignment. The attention mechanism-based method also has a large amount of calculation. Similar to the image restoration method based on deep learning, although it has better performance, it also has problems such as lack of interpretability due to the "black box" operation of deep learning and failure to fully utilize relevant knowledge in the field of video restoration. In addition, due to the high dimensionality of the video, the above problems will further lead to more complex network design and difficulty in training and convergence of the network.

[0074] In view of the above problems, the present invention provides a method and system for video restoration guided by deep prior optical flow. The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0075] In order to illustrate the deep prior optical flow guided video restoration method and system provided by the present invention, Figure 1-Figure 4 The embodiments of the present invention are exemplarily indicated.

[0076] The following description of exemplary embodiments is merely illustrative in nature and is not intended to limit the present invention and its application or use. Technologies and devices known to ordinary technicians in the relevant field may not be discussed in detail, but in appropriate cases, the technologies and devices should be considered as part of the specification.

[0077] like Figure 1 , Figure 2 , Figure 3 As shown together, the deep prior optical flow guided video restoration method provided by the present invention includes:

[0078] S1: Input the target video into a deep a priori video framework network with a fixed number of stages to iterate the target video in stages, each stage iteration corresponds to a classic semi-quadratic splitting algorithm, and in each iteration process, a preset data module extracts and outputs data for the target video, a preset reconstruction module locks the video to be repaired in the target video, and performs video repair based on optical flow guiding local attention on the video to obtain a repaired video.

[0079] Specifically, in a specific embodiment, the deep prior optical flow guided video restoration method in this embodiment is applied to a server loaded with Ubuntu 20.04.4 system. The deep prior video framework network with a fixed number of stages uses Pytorch 1.12.1, and the corresponding Python version is 3.7. The training phase uses 8 NVIDIA GeForce RTX3090 graphics cards for GPU acceleration, and the testing phase is performed on a single NVIDIA GeForce RTX 3090 graphics card.

[0080] Specifically, the deep prior video framework network with a fixed number of stages in this embodiment is expanded and optimized in combination with a traditional model-based optimization algorithm. Specifically, video restoration is regarded as an ill-posed inverse problem, and from the perspective of combining the advantages of model-based and deep learning-based methods, the update steps of the classic semi-quadratic splitting algorithm for solving optimization problems are mapped to a network architecture with a fixed number of stages, each stage corresponds to an iteration of the classic semi-quadratic splitting algorithm, and consists of a group of modules that solve the data items and prior items in one iteration respectively.

[0081] That is, each stage iteration corresponds to a classic semi-quadratic splitting algorithm. The semi-quadratic splitting algorithm is used to solve the optimization problem. By introducing an auxiliary variable Z, the formula The unconstrained optimization problem described is transformed into a constrained optimization problem:

[0082] in, Represents the original undamaged video, is a frame-by-frame binary mask indicating the missing area in the video, ⊙ represents the point-by-point multiplication of the corresponding elements, The damaged video (damaged target video) is obtained by multiplying the original video and the mask frame by frame. Represents the restored video. Formula It can be reformulated as the following optimization problem:

[0083]

[0084] Among them, μ is a penalty parameter

[0085] The semi-quadratic splitting algorithm iteratively solves between the following two equations

[0086]

[0087] Where k is the iteration index of the semi-quadratic splitting algorithm. μ should be large enough and increase iteratively. The μ used in the kth iteration is μ k express.

[0088] The data term and the prior term are decoupled into two separate sub-problems. It is related to the quadratic regularized least squares problem, whose closed form solution is as follows:

[0089]

[0090] where α k-1 =μ k-1 σ 2 ,formula The prior term Φ(X) in Φ(X) is related to the original uncorrupted video, but in practice can only be estimated from the corrupted video.

[0091] The formula and The update steps of the semi-quadratic splitting algorithm are mapped into a network architecture consisting of multiple stages, each of which corresponds to an iteration in the semi-quadratic splitting algorithm. Each stage is solved by the formula It consists of a data module and a reconstruction module for learning deep priors.

[0092] During each iteration, a preset data module extracts and outputs data from the target video, a preset reconstruction module locks the video to be repaired in the target video, and performs video repair based on optical flow-guided local attention on the video to obtain a repaired video.

[0093] To prove the effectiveness of the design of this embodiment, two mainstream datasets commonly used in the field of video restoration are used, namely the DAVIS dataset and the Youtube-VOS dataset, to compare the proposed video restoration method with representative and advanced deep learning-based video restoration methods VINet, DFVI, LGTSM, CAP, FGVC, STTN, FuseFormer, E2FGVI, and other methods mentioned above. Stable masks and object masks are used for simulation respectively. PSNR, SSIM, VFID and Ewarp are selected as indicators for comparison; the comparative experimental results on the YouTube-VOS dataset and the comparison results of the computational amount and running time are shown in Table 1. As shown in Table 1, compared with all other comparative methods, the best results are achieved in the four performance evaluation indicators of PSNR, SSIM, VFID and Ewarp: the average PSNR is 4.91dB higher than the lowest DFVI, and 0.36dB higher than the second-best E2FGVI and the method proposed in Chapter 3; the average SSIM is 0.0311 higher than the lowest FGVC, and 0.0014 higher than the second-best E2FGVI; the average VFID is 0.028 lower than the highest VINet, and 0.002 lower than the second-best E2FGVI; the average Ewarp is 0.1045×10 lower than the highest LGTSM -2 , which is 0.0042×10 lower than the second best method proposed in Chapter 3. -2 .

[0094] Table 1:

[0095]

[0096]

[0097] Therefore, by combining traditional methods and methods based on deep learning, the performance of the network is improved, while being interpretable and reliable. The comparative experimental results show that the deep prior-based video restoration method implemented in this embodiment is superior to the existing advanced video restoration methods.

[0098] The steps of video restoration based on optical flow guided local attention to obtain the restored video include:

[0099] S11: dividing the video to be repaired into local continuous adjacent frames and global reference frames, and converting the local continuous adjacent frames into local continuous adjacent features and converting the global reference frames into global reference features through a preset encoder;

[0100] S12: using a preset optical flow completion module to predict and complete the local continuous adjacent frames between each other to obtain a bidirectional optical flow and a bidirectional optical flow occlusion mask, and obtaining a supplemented aligned local continuous adjacent feature based on the bidirectional optical flow and the local continuous adjacent feature;

[0101] S13: using a preset inter-frame restoration module to obtain a global time information video feature according to the local continuous adjacent features after the complementary alignment and the global reference features;

[0102] S14: Decode the global time information video feature by a preset decoder to obtain a restored video.

[0103] In addition, after obtaining the repaired video, step S15 can be performed to use a temporal PatchGAN discriminator to determine whether the repaired video is similar to the real scene distribution, so as to improve the video quality.

[0104] Specifically, in step S11, the video to be repaired is divided into local continuous adjacent frames and global reference frames, and the local continuous adjacent frames are converted into local continuous adjacent features and the global reference frames are converted into global reference features by a preset encoder. In a specific embodiment, given an original video with a sequence length of T and a frame-by-frame binary mask indicating the corresponding missing regions Input damaged video frame (damaged part of the video to be repaired) It can be obtained by multiplying the original video and the corresponding mask point by point according to Y=M⊙X frame by frame.

[0105] First, all input video frames are fed into the encoder. A context encoder (·) can be used to obtain low-resolution features with a size of 1 / d of the original video frame (segment of the target video) through multiple downsampling operations. To improve the computational efficiency of subsequent processing: E ​​= encoder (X).

[0106] Then, step S12 is performed: a preset optical flow completion module is used to predict and complete the local continuous adjacent frames between each other to obtain a bidirectional optical flow and a bidirectional optical flow occlusion mask, and based on the bidirectional optical flow and the local continuous adjacent features, a supplemented aligned local continuous adjacent feature is obtained; wherein the bidirectional optical flow includes a completed forward optical flow and a completed backward optical flow; that is, for the local continuous adjacent frames First, an optical flow completion module flow(·) is used to obtain the forward optical flow between adjacent frames after d times downsampling. Backward Optical Flow The occlusion mask corresponding to the two markings that indicate pixel visibility in the optical flow and

[0107] The optical flow completion module is a pre-trained module; wherein the step of training the optical flow completion module comprises:

[0108] Inputting the sample loss video into the optical flow estimation network SpyNet, so that the optical flow estimation network SpyNet obtains and completes the forward optical flow and the backward optical flow of the sample loss video, and learns the sample occlusion mask with pixel visibility in the forward optical flow and the backward optical flow;

[0109] The optical flow estimation network SpyNet is optimized based on the sample occlusion mask and the preset occlusion mask of the complete video sample using a preset loss function to obtain an optical flow network until the data of the loss function reaches a preset threshold, and the optical flow network trained for the last time is used as an optical flow completion module.

[0110] These optical flows are used to guide the network to propagate the valid features extracted in the known area to the damaged area across frames, and the occlusion mask is used to alleviate the ambiguity caused by the optical flow warping operation in the presence of occlusion or rapid motion of objects in the video.

[0111] The optical flow completion module trained in this way can automatically obtain the repair result.

[0112] Specifically, for each frame feature to be repaired, a many-to-one approach is used to propagate all remaining local neighboring frame features to the current frame to obtain the repair result:

[0113]

[0114] Where propagation(·) represents many-to-one feature propagation, Represents the set of bidirectional optical flows between adjacent frames involved. In this process, feature alignment based on optical flow to guide local attention is required:

[0115]

[0116] Among them, align(·) represents the feature alignment based on optical flow to guide local attention, I in represents the set of all remaining local neighboring frame features, Represents the set of all forward and backward optical flows involved.

[0117] Rough alignment through optical flow can determine a spatiotemporal feature block of a certain size centered on the pixels in the missing area, which is used as a rough search range. Local multi-head attention is used to calculate weights to complete further alignment and fusion of features, while reducing the impact of optical flow estimation errors.

[0118] Then, in step S13, a preset inter-frame restoration module is used to obtain a global time information video feature according to the local continuous adjacent features after the complementary alignment and the global reference features; wherein the step of obtaining a global time information video feature according to the local continuous adjacent features after the complementary alignment and the global reference features includes:

[0119] Performing time axis alignment on the local continuous adjacent features after the complementary alignment and the global reference features;

[0120] In the same time period, comparing the local continuous adjacent features after the supplementary alignment with the global reference features to obtain video status parameters;

[0121] If the video status parameter indicates that the local continuous adjacent features after the supplementary alignment are better than the global reference features, then in the same time period, the global reference features are replaced by the local continuous adjacent features after the supplementary alignment to form the global time information video features.

[0122] In a specific embodiment, a Temporal focal transformer can be introduced as an inter-frame restoration module tft(·) to transform all local continuous adjacent frame features that have undergone feature propagation and alignment into a single pixel. and global reference frame features Input, combine global and local information for refinement and completion:

[0123] In this way, the global temporal information video features are obtained.

[0124] Finally, the global time information video features are decoded by a preset decoder to obtain the repaired video, that is, the obtained global time information video features are input into the decoder decoder(·), and the final repair result is obtained after upsampling.

[0125] Thus, the restored video was obtained.

[0126] The temporal PatchGAN discriminator proposed by Chang et al. can then be used to determine whether the restored video is close to the real scene distribution, thus ensuring the authenticity of the video restoration.

[0127] In order to further verify the effect of this embodiment, the relevant data in Table 1 are taken together, and Table 2 is obtained based on the comparative experimental results on the DAVIS dataset. As shown in Table 2, the method proposed in this embodiment achieves almost the best results in the four performance evaluation indicators of PSNR, SSIM, VFID and Ewarp compared with all the comparative methods: the average PSNR is 4.8dB higher than that of LGTSM with the lowest value, and 0.36dB higher than that of E2FGVI with the second best performance and the method proposed in Chapter 3; the average SSIM is 0.0332 higher than that of DFVI with the lowest value, and 0.001 5 higher than that of E2FGVI with the second best performance and the method proposed in Chapter 3; the average VFID is 0.09 lower than that of VINet with the highest value, and 0.007 lower than that of E2FGVI with the second best performance; the performance on average Ewarp is slightly inferior to that of the method proposed in Chapter 3, which is 0.0016×10-1 higher, but 0.0034×10-1 lower than that of E2FGVI with the best performance among the remaining methods. Although the video restoration method based on optical flow guided local attention only achieved results comparable to E2FGVI, the video restoration method based on deep prior improved on it and simplified the optical flow completion module, ultimately achieving results that exceeded E2FGVI, which fully demonstrates the effectiveness of the proposed combined model-driven and data-driven method.

[0128] Table 2:

[0129]

[0130]

[0131] For the qualitative results under the stable mask setting, other existing methods have problems such as insufficient understanding of scene semantics, blurring at the junction of completed objects, and insufficient and accurate texture structure details. Compared with the comparison method, the video frames repaired by the deep prior optical flow guided video restoration method in this embodiment have more accurate content and more comprehensive restoration of details, which is closest to the original video. The restoration results show the effectiveness of the network framework proposed in this chapter. In addition, for the qualitative results under the object mask setting, other existing methods have problems such as insufficient coherent and smooth texture structure, large areas of blurring in the area to be repaired, and lack of details. The deep prior optical flow guided video restoration method in this embodiment combines optical flow and local attention to effectively utilize and maintain the spatiotemporal consistency of the video and synthesize content that is more reasonable in terms of semantics and texture.

[0132] As described above, the deep a priori optical flow guided video restoration method provided by the present invention first inputs the target video into a deep a priori video framework network with a fixed number of stages to iterate the target video in stages, each stage iteration corresponds to a classic semi-quadratic splitting algorithm, and in each iteration process, a preset data module extracts and outputs data from the target video, a preset reconstruction module locks the video to be restored in the target video, and performs video restoration based on optical flow guided local attention on the video to be restored to obtain a restored video; wherein, the video to be restored is first divided into local continuous adjacent frames and global reference frames, and the local continuous adjacent frames are converted into local continuous adjacent features through a preset encoder, and the global reference frames are converted into global reference features, and then a preset optical flow restoration is used. The full module predicts and completes the local continuous adjacent frames between each other to obtain bidirectional optical flow and bidirectional optical flow occlusion masks, and obtains the supplemented aligned local continuous adjacent features based on the bidirectional optical flow and the local continuous adjacent features. Then, a preset inter-frame repair module is used to obtain the global time information video features according to the supplemented aligned local continuous adjacent features and the global reference features. Finally, the global time information video features are decoded by a preset decoder to obtain the repaired video. In this way, the traditional model-driven method and the data-driven method based on deep learning are combined to form a deep prior video framework with a fixed number of stages. The combination of traditional methods and deep learning-based methods improves network performance, improves interpretability and reliability, reduces network design structure, reduces computational overhead, and improves video repair efficiency.

[0133] like Figure 4 As shown, the present invention also provides a deep prior optical flow guided video restoration system 100, which implements the deep prior optical flow guided video restoration method as described above, including: a deep prior video framework network 110 with a fixed number of stages, which is used to receive a target video and iterate the target video in stages, and each stage iteration corresponds to a classic semi-quadratic splitting algorithm;

[0134] The deep a priori video framework network includes a data module 111 and a reconstruction module 112; the data module 111 is used to extract and output data from the target video in each iteration process; the reconstruction module 112 is used to lock the video to be repaired in the target video, and perform video repair based on optical flow to guide local attention on the video to be repaired to obtain a repaired video;

[0135] The reconstruction module 112 includes:

[0136] The video data processing module 1121 is used to divide the video to be repaired into local continuous adjacent frames and global reference frames;

[0137] Encoder 1122, configured to convert the local continuous adjacent frames into local continuous adjacent features, and convert the global reference frame into a global reference feature;

[0138] An optical flow completion module 1123 is used to predict and complete local continuous adjacent frames between each other to obtain bidirectional optical flow and bidirectional optical flow occlusion mask;

[0139] An optical flow guided feature propagation and alignment module 1124, configured to obtain supplementary aligned local continuous adjacent features based on the bidirectional optical flow and the local continuous adjacent features;

[0140] An inter-frame restoration module 1125 is used to obtain a global time information video feature according to the local continuous adjacent features after the complementary alignment and the global reference features;

[0141] The decoder 1126 is used to decode the global time information video feature to obtain a restored video.

[0142] The optical flow completion module is a pre-trained module; wherein the step of training the optical flow completion module comprises:

[0143] Inputting the sample loss video into the optical flow estimation network SpyNet, so that the optical flow estimation network SpyNet obtains and completes the forward optical flow and the backward optical flow of the sample loss video, and learns the sample occlusion mask with pixel visibility in the forward optical flow and the backward optical flow;

[0144] The optical flow estimation network SpyNet is optimized based on the sample occlusion mask and the preset occlusion mask of the complete video sample using a preset loss function to obtain an optical flow network until the data of the loss function reaches a preset threshold, and the optical flow network trained for the last time is used as an optical flow completion module.

[0145] The optical flow-guided feature propagation and alignment module includes a guided alignment module and a local multi-head attention mechanism.

[0146] The guided alignment module is used to select an undamaged frame from the local continuous adjacent frames as a reference frame; wrap the reference frame around the current frame to be repaired for rough alignment through optical flow convolution, and lock the damaged optical flow gap of the current frame to be repaired based on the local continuous adjacent features; search for an occlusion mask with pixel visibility corresponding to the damaged optical flow gap in the bidirectional optical flow;

[0147] The local multi-head attention mechanism is used to obtain the local attention of the occlusion mask, and replace the occlusion mask with strong local attention on the local continuous adjacent features corresponding to the damaged optical flow gap to obtain the local continuous adjacent features after supplementary alignment.

[0148] The step of obtaining the global time information video feature by the inter-frame repair module according to the local continuous adjacent features after the supplementary alignment and the global reference features comprises:

[0149] The step of acquiring the global time information video feature according to the local continuous adjacent features after the complementary alignment and the global reference feature comprises:

[0150] Performing time axis alignment on the local continuous adjacent features after the complementary alignment and the global reference features;

[0151] In the same time period, comparing the local continuous adjacent features after the supplementary alignment with the global reference features to obtain video status parameters;

[0152] If the video status parameter indicates that the local continuous adjacent features after the supplementary alignment are better than the global reference features, then in the same time period, the global reference features are replaced by the local continuous adjacent features after the supplementary alignment to form the global time information video features.

[0153] The specific implementation method refers to the above-mentioned embodiment of the deep prior optical flow guided video restoration method, which will not be described in detail here.

[0154] As described above, the deep a priori optical flow guided video restoration system provided by the present invention divides the video to be restored into local continuous adjacent frames and global reference frames through the video data processing module 1121; converts the local continuous adjacent frames into local continuous adjacent features through the encoder 1122, and converts the global reference frame into a global reference feature; predicts and completes the local continuous adjacent frames between each other through the optical flow completion module 1123 to obtain a bidirectional optical flow and a bidirectional optical flow occlusion mask; obtains the supplemented aligned local continuous adjacent features based on the bidirectional optical flow and the local continuous adjacent features through the optical flow guided feature propagation and alignment module 1124; obtains the global time information video features based on the supplemented aligned local continuous adjacent features and the global reference features through the inter-frame restoration module 1125; and decodes the global time information video features through the decoder 1126 to obtain a restored video, thereby combining the traditional model-driven method with the data-driven method based on deep learning to form a deep a priori video framework with a fixed number of stages, combining the traditional method with the method based on deep learning to improve network performance, improve interpretability and reliability, and reduce the network design structure, reduce computational overhead, and improve video restoration efficiency.

[0155] As described above with reference to the accompanying drawings, the method and system for video restoration guided by optical flow with depth prior proposed by the present invention are described by way of example. However, those skilled in the art should understand that various improvements can be made to the method and system for video restoration guided by optical flow with depth prior proposed by the present invention without departing from the content of the present invention. Therefore, the protection scope of the present invention should be determined by the content of the attached claims.

Claims

1. A deep prior optical flow guided video restoration method, It is characterized in that include: The target video is input into a deep a priori video framework network with a fixed number of stages to iterate the target video in stages, each stage iteration corresponds to a classic semi-quadratic splitting algorithm, and in each iteration process, a preset data module extracts and outputs data from the target video, a preset reconstruction module locks the video to be repaired in the target video, and performs video repair based on optical flow-guided local attention on the video to obtain a repaired video; wherein, When the classic semi-quadratic splitting algorithm is used to optimize the target video, the semi-quadratic splitting algorithm is used to solve the optimization problem. By introducing an auxiliary variable Z, the formula The unconstrained optimization problem described is transformed into a constrained optimization problem: in, Represents the original undamaged video, is a frame-by-frame binary mask indicating the missing area in the video, ⊙ represents the point-by-point multiplication of the corresponding elements, The damaged target video is obtained by multiplying the original video and the mask frame by frame. represents the restored video; and the formula It can be reformulated as the following optimization problem: Among them, μ is a penalty parameter; The semi-quadratic splitting algorithm iteratively solves between the following two equations: Where k represents the iteration index of the semi-quadratic splitting algorithm; μ is set to the maximum, μ is increased in an iterative manner, and μ used in the kth iteration is replaced by μ k express; formula It is related to the quadratic regularized least squares problem, whose closed form solution is as follows: where α k-1 =μ k-1 σ 2 ,formula The prior term Φ(X) in Φ(X) is related to the original uncorrupted video and can only be estimated from the corrupted video in practice. The formula and The update steps of the semi-quadratic splitting algorithm are mapped into a network architecture consisting of multiple stages, each of which corresponds to an iteration in the semi-quadratic splitting algorithm. Each stage is solved by the formula It consists of a data module and a reconstruction module for learning deep priors; The steps of video restoration based on optical flow guided local attention to obtain the restored video include: The video to be repaired is divided into local continuous adjacent frames and global reference frames, and the local continuous adjacent frames are converted into local continuous adjacent features and the global reference frames are converted into global reference features through a preset encoder; Using a preset optical flow completion module to predict and complete the local continuous adjacent frames between each other to obtain a bidirectional optical flow and a bidirectional optical flow occlusion mask, and obtaining a supplemented aligned local continuous adjacent feature based on the bidirectional optical flow and the local continuous adjacent feature; Using a preset inter-frame restoration module to obtain a global time information video feature according to the local continuous adjacent features after the complementary alignment and the global reference features; The global time information video feature is decoded by a preset decoder to obtain a restored video.

2. The deep prior optical flow guided video restoration method as claimed in claim 1, It is characterized in that The optical flow completion module is a pre-trained module; wherein the step of training the optical flow completion module comprises: Inputting the sample loss video into the optical flow estimation network SpyNet, so that the optical flow estimation network SpyNet obtains and completes the forward optical flow and the backward optical flow of the sample loss video, and learns the sample occlusion mask with pixel visibility in the forward optical flow and the backward optical flow; The optical flow estimation network SpyNet is optimized based on the sample occlusion mask and the preset occlusion mask of the complete video sample using a preset loss function to obtain an optical flow network until the data of the loss function reaches a preset threshold, and the optical flow network trained for the last time is used as an optical flow completion module.

3. The deep prior optical flow guided video restoration method as claimed in claim 2, It is characterized in that The bidirectional optical flow includes a completed forward optical flow and a completed backward optical flow.

4. The deep prior optical flow guided video restoration method as claimed in claim 3, It is characterized in that The step of acquiring the supplementally aligned local continuous adjacent features based on the bidirectional optical flow and the local continuous adjacent features comprises: Selecting an undamaged frame from the local continuous adjacent frames as a reference frame; The reference frame is wrapped around the current frame to be repaired by optical flow convolution for rough alignment, and the damaged optical flow gap of the current frame to be repaired is locked based on the local continuous adjacent features; searching the bidirectional optical flow for an occlusion mask with pixel visibility corresponding to the damaged optical flow gap; The occlusion mask with strong local attention is replaced on the local continuous neighboring features corresponding to the damaged optical flow gap to obtain the local continuous neighboring features after complementary alignment.

5. The deep prior optical flow guided video restoration method as claimed in claim 4, It is characterized in that The step of acquiring the global time information video feature according to the local continuous adjacent features after the complementary alignment and the global reference feature comprises: Performing time axis alignment on the local continuous adjacent features after the complementary alignment and the global reference features; In the same time period, comparing the local continuous adjacent features after the supplementary alignment with the global reference features to obtain video status parameters; If the video status parameter indicates that the local continuous adjacent features after the supplementary alignment are better than the global reference features, then in the same time period, the global reference features are replaced by the local continuous adjacent features after the supplementary alignment to form the global time information video features.

6. A deep prior optical flow guided video restoration system, It is characterized in that Implementing the deep a priori optical flow guided video restoration method as described in claims 1-5, comprising a deep a priori video framework network with a fixed number of stages, for receiving a target video, and iterating the target video in stages, each stage iteration corresponding to a classic semi-quadratic splitting algorithm; The deep a priori video framework network includes a data module and a reconstruction module; the data module is used to extract and output data from the target video in each iteration process; the reconstruction module is used to lock the video to be repaired in the target video, and perform video repair based on optical flow to guide local attention on the video to be repaired to obtain a repaired video; wherein, When the classic semi-quadratic splitting algorithm is used to optimize the target video, the semi-quadratic splitting algorithm is used to solve the optimization problem. By introducing an auxiliary variable Z, the formula The unconstrained optimization problem described is transformed into a constrained optimization problem: in, Represents the original undamaged video, is a frame-by-frame binary mask indicating the missing area in the video, ⊙ represents the point-by-point multiplication of the corresponding elements, The damaged target video is obtained by multiplying the original video and the mask frame by frame. represents the restored video; and the formula It can be reformulated as the following optimization problem: Among them, μ is a penalty parameter; The semi-quadratic splitting algorithm iteratively solves between the following two equations: Where k represents the iteration index of the semi-quadratic splitting algorithm; μ is set to the maximum, μ is increased in an iterative manner, and μ used in the kth iteration is replaced by μ k express; formula It is related to the quadratic regularized least squares problem, whose closed form solution is as follows: where α k-1 =μ k-1 σ 2 ,formula The prior term Φ(X) in Φ(X) is related to the original uncorrupted video and can only be estimated from the corrupted video in practice. The formula and The update steps of the semi-quadratic splitting algorithm are mapped into a network architecture consisting of multiple stages, each of which corresponds to an iteration in the semi-quadratic splitting algorithm. Each stage is solved by the formula It consists of a data module and a reconstruction module for learning deep priors; The reconstruction module comprises: A video data processing module, used for dividing the video to be repaired into local continuous adjacent frames and global reference frames; An encoder, configured to convert the local continuous adjacent frames into local continuous adjacent features, and convert the global reference frame into a global reference feature; The optical flow completion module is used to predict and complete the local continuous adjacent frames between each other to obtain the bidirectional optical flow and bidirectional optical flow occlusion mask; An optical flow-guided feature propagation and alignment module, used for acquiring supplementary aligned local continuous adjacent features based on the bidirectional optical flow and the local continuous adjacent features; An inter-frame restoration module, used for acquiring a global time information video feature according to the local continuous adjacent features after the complementary alignment and the global reference features; A decoder is used to decode the global time information video feature to obtain a restored video.

7. The deep prior optical flow guided video inpainting system as claimed in claim 6, It is characterized in that The optical flow completion module is a pre-trained module; wherein the step of training the optical flow completion module comprises: Inputting the sample loss video into the optical flow estimation network SpyNet, so that the optical flow estimation network SpyNet obtains and completes the forward optical flow and the backward optical flow of the sample loss video, and learns the sample occlusion mask with pixel visibility in the forward optical flow and the backward optical flow; The optical flow estimation network SpyNet is optimized based on the sample occlusion mask and the preset occlusion mask of the complete video sample using a preset loss function to obtain an optical flow network until the data of the loss function reaches a preset threshold, and the optical flow network trained for the last time is used as an optical flow completion module.

8. The deep prior optical flow guided video inpainting system as claimed in claim 7, It is characterized in that The optical flow-guided feature propagation and alignment module includes a guided alignment module and a local multi-head attention mechanism.

9. The deep prior optical flow guided video inpainting system as claimed in claim 8, It is characterized in that The guided alignment module is used to select an undamaged frame from the local continuous adjacent frames as a reference frame; The reference frame is wrapped around the current frame to be repaired by optical flow convolution for rough alignment, and the damaged optical flow gap of the current frame to be repaired is locked based on the local continuous adjacent features; an occlusion mask with pixel visibility corresponding to the damaged optical flow gap is searched in the bidirectional optical flow; The local multi-head attention mechanism is used to obtain the local attention of the occlusion mask, and replace the occlusion mask with strong local attention on the local continuous adjacent features corresponding to the damaged optical flow gap to obtain the local continuous adjacent features after supplementary alignment.

10. The deep prior optical flow guided video inpainting system as claimed in claim 9, It is characterized in that The step of obtaining the global time information video feature by the inter-frame repair module according to the local continuous adjacent features after the supplementary alignment and the global reference features comprises: The step of acquiring the global time information video feature according to the local continuous adjacent features after the complementary alignment and the global reference feature comprises: Performing time axis alignment on the local continuous adjacent features after the complementary alignment and the global reference features; In the same time period, comparing the local continuous adjacent features after the supplementary alignment with the global reference features to obtain video status parameters; If the video status parameter indicates that the local continuous adjacent features after the supplementary alignment are better than the global reference features, then in the same time period, the global reference features are replaced by the local continuous adjacent features after the supplementary alignment to form the global time information video features.