Video inpainting method based on multi-module stepwise fusion learning

Abstract

The application discloses a multi-module stepwise fusion learning video repairing method, a phased training process is used for screening out part of scratches by using a scratch detection module, and optimization is carried out through a function optimization module to obtain optimal parameters θ U ; the function optimization module is used for optimizing the scratch detection module alone; the function optimization module is used for optimizing the optical flow transformation fusion module and the mask extraction module together; an optimal overall parameter is obtained through a fusion optimization process; and the inference process uses the optimal parameters θ obtained in the fusion optimization process as parameters of the scratch detection module, the optical flow transformation fusion module, the mask extraction module and the multi-module fusion module all , the video or image sequence to be processed is converted into a data form suitable for a model{frame t}, an output frame y fusion is obtained after processing, and the output frame y is converted into an actual output video sequence.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of video processing, and particularly relates to a multi-module stepwise fusion learning video restoration method. BACKGROUND

[0002] In the film digitization process, scratches or stains caused by various factors in the whole process from film production to digitization will leave defects in the picture after being digitized into digital video, thereby affecting the viewing effect, so removing these defects is the key to preserving the value of the film.

[0003] At present, for scratch removal, the traditional image processing method is to detect the position of the scratch, and then use image similar blocks for filling. The above processing method has many problems. First, the accuracy of detection is not enough, and second, the area directly filled with image blocks will be quite different from the overall image, and the edge boundary is obvious.

[0004] And for the repair of irregular areas, the traditional image processing method is helpless, and can only use the deep learning method. At present, there is a method based on three-dimensional convolution, which uses neural network training to remove small area spots and stains, but when a larger motion displacement object appears in the video, external noise will be introduced, resulting in abnormal processing and poor effect. SUMMARY

[0005] In view of the above problems, the application provides a multi-module stepwise fusion learning video restoration method, which is used for adopting the method of training in stages and then fusing and optimizing, obtaining sub-modules for repairing different types of defects by training in stages, and then performing feature fusion and optimization, so as to obtain more accurate parameters and achieve the purpose of repairing the video.

[0006] To solve the above technical problems, the application adopts the following technical scheme:

[0007] A multi-module stepwise fusion learning video restoration method, comprising a stage-by-stage training process, a fusion and optimization process and an inference process,

[0008] The stage-by-stage training process is used to screen out a part of scratches by using a scratch detection module, input a training data pair <x_input t , y_mask t > representing the to-be-processed data and the real scratch type feature, output a group of screened scratch type features y_mask detect , and obtain the best parameters theta by optimizing the function optimization module U ; and the optical flow transformation fusion module inputs a training data sequence pair <x_input t-i , x_input t,yt>, where x_input t-i x_input represents the i-th frame to be processed. t Indicates the current frame to be processed, y t Representing the true reference mask, the optical flow transformation (Flow) requires pairwise operations between the current frame and the preceding and following frames. First, the optical flow information between the two frames is calculated, then an affine transformation (Warp) operation is performed. Finally, the fully transformed and fused sequence is output and given to the mask extraction module. The mask extraction module spatially aligns the output sequence of the optical flow transformation fusion module and fuses the temporal information of the preceding and following frames with the current frame to obtain accurate mask information y. extract The scratch detection module is optimized separately using a function optimization module. The optimization target is the true scratch type y_mask. t and the selected scratch type feature y_mask detect The minimum reconstruction loss is calculated between the two to find the optimal parameter θ. U The optical flow transformation fusion module and the mask extraction module are optimized together using a function optimization module. The optimization target is the real reference mask y. t and extracted mask information y extract Calculate the minimum reconstruction loss to obtain the optimal parameter θ. F ;

[0009] The fusion optimization process involves the scratch detection module, optical flow transformation fusion module, and mask extraction module each using the optimal parameters θ obtained during the phased training process. U With θ F The parameters of the scratch detection module, optical flow transformation fusion module, mask extraction module, and multi-module fusion module are optimized as a whole through the function optimization module. This is achieved by calculating the reference image gt and the fused data y_output. fusion Minimize the reconstruction loss between the optimal overall parameters θ. all ;

[0010] The inference process uses the optimal parameter θ obtained during the fusion optimization process for the scratch detection module, optical flow transformation fusion module, mask extraction module, and multi-module fusion module. all This converts the video or image sequence to be processed into a data format suitable for the model, {frame}. t}, the processed output frame y fusion Converted into the actual output video sequence.

[0011] In one possible implementation, the scratch detection module includes multiple convolutional operations (Conv) and activation functions. The activation functions are of two types, denoted as follows:

[0012] α is a learnable parameter

[0013]

[0014] The scratch detection module structure is a U-shaped pyramid structure, that is, each layer includes a convolution operation and an activation function, the number of layers is not less than six, the operation transition between the front half of the layers uses 2 times down sampling, and the operation transition between the rear half uses 2 times up sampling, and the whole expression is U(x), the input of the scratch detection module is a training data pair <x_input t , y_mask t > to be processed, the output is a group of screened scratch type features y_mask detect , and the scratch detection module is optimized by the function optimization module to obtain optimal parameters θ U .

[0015] In a possible implementation, the function optimization module is used to optimize the scratch detection module alone, the optimization target is the real scratch type y_mask t and the screened scratch type feature y_mask detect , and the minimum reconstruction loss is calculated between the two, so as to find the formula expression of the optimal parameters θ U .

[0016] L = ∑ (y_mask t -y_mask detect ) 2

[0017]

[0018] In a possible implementation, the function optimization module is used to optimize the optical flow transformation fusion module and the mask extraction module together, the optimization target is the real reference mask y t and the extracted mask information y extract , the minimum reconstruction loss is calculated, and the formula expression of the optimal parameters θ F .

[0019] L = ∑ (y t -y extract ) 2

[0020]

[0021] In a possible implementation, in the inference process, the data inflow module is used to convert the video or image sequence to be processed into a data form suitable for the model {frame t} represents all frames, the number of input frames can be increased or reduced as needed, and the data is transmitted to the scratch detection module and the optical flow transformation fusion module after processing.

[0022] The present invention offers the following advantages: The technical solution employs a phased training followed by fusion optimization approach. Phased training yields sub-modules for repairing different types of defects, followed by feature fusion optimization to obtain more accurate parameters, thus achieving the goal of video restoration. The phased training of different modules in the technical solution of this invention makes the model highly modular and more robust; the processed sequence does not suffer from artifacts caused by large motions, improving the restoration quality. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the multi-module, step-by-step fusion learning video restoration method according to an embodiment of the present invention.

[0024] Figure 2 This is a comparison chart showing the effect of a specific application example after video restoration using the multi-module step-by-step fusion learning method of this invention.

[0025] Figure 3 This is a comparison chart showing the effect of another specific application example of the video restoration method using the multi-module step-by-step fusion learning method of this invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] See Figure 1 The diagram shows a flowchart of a multi-module, step-by-step fusion learning video restoration method according to the present invention, including a phased training process, a fusion optimization process, and an inference process.

[0028] The phased training process is used to initially screen out a portion of scratches using the scratch detection module, and then input the training data to be processed. <x_input t y_mask t > represents the data to be processed and the actual scratch type features, and outputs a set of filtered scratch type features y_mask. detect The optimal parameters θ are obtained through optimization using the function optimization module. U The optical flow transform fusion module inputs training data sequence pairs. <x_input t-i x_input t y t >, where x_inputt-i represents the i-th frame before processing, x_input t represents the current frame to be processed, y t represents the real reference mask, the optical flow transformation Flow needs to operate two frames respectively with the current frame two by two, first calculate the optical flow information between the two frames, then do the affine transformation operation Warp, finally output the complete transformed and fused sequence to the mask extraction module; the mask extraction module performs spatial alignment on the output sequence of the optical flow transformation and fusion module, fuses the time sequence information of the front and rear frames and the current frame, and obtains accurate mask information y extract ; the function optimization module is used to optimize the scratch detection module alone, and the optimization target is the real scratch type y_mask t and the selected scratch type feature y_mask detect , the minimum reconstruction loss is calculated between the two, so as to find the best parameter θ U ; the function optimization module is used to optimize the optical flow transformation and fusion module and the mask extraction module together, and the optimization target is the real reference mask y t and the extracted mask information y extract , the minimum reconstruction loss is calculated, and the best parameter θ F is obtained.

[0029] The fusion optimization process uses the best parameters θ U and θ F obtained in the stage-by-stage training process for the scratch detection module, the optical flow transformation and fusion module, and the mask extraction module, respectively. fusion The function optimization module is used to optimize the parameters of the scratch detection module, the optical flow transformation and fusion module, the mask extraction module and the multi-module fusion module as a whole, and the best overall parameter θ all is obtained by calculating the minimum reconstruction loss between the reference image gt and the fused data y_output all .

[0030] In the inference process, the parameters of the scratch detection module, the optical flow transformation and fusion module, the mask extraction module and the multi-module fusion module use the best parameters θ all obtained in the fusion optimization process. t The video or image sequence to be processed is converted into a data form suitable for the model {frame fusion}, the output frame y fusion after processing is converted into the actual output video sequence.

[0031] The video repair method of the multi-module step-by-step fusion learning set above adopts the data inflow module to unify the input data of various format types, so that the scratch detection module and the optical flow transformation fusion module can identify the format; the data outflow module is adopted to convert the data output by the multi-stage fusion module into a format consistent with the input data of the inference process; the scratch detection module is responsible for screening out part of the scratches, the scratch type detected by this module is limited, and the detection result is handed over to the multi-module fusion module and the function optimization module; the optical flow transformation fusion module is adopted to calculate the optical flow between the input image and the front and rear multiple frames of sequences, and after transformation and fusion, it is handed over to the mask extraction module and the function optimization module; the mask extraction module uses the difference between the current frame and the surrounding frames in the time sequence to find out the defect area of the current frame according to the feature information after the optical flow transformation fusion, and uses the mask to represent, and finally the mask is sent to the multi-module fusion module and the function optimization module; the multi-module fusion module is responsible for splicing the results output by the scratch detection module and the mask extraction module, fusing the results with the current processing frame and the surrounding frames, and finally outputting the processing result to the function optimization module for parameter updating, and outputting to the data outflow module after optimization; the function optimization module is adopted to first train the parameters of the scratch detection module, the optical flow transformation fusion module and the mask extraction module in stages, and then optimize all the modules that need to be optimized, and finally all the optimized modules are used for the inference process. The way of training in stages and then optimizing in fusion is adopted, the sub-modules for repairing different types of defects are obtained by training in stages, and then the feature fusion optimization is performed, so that more accurate parameters are obtained, and the purpose of repairing the video is achieved.

[0032] In an embodiment of the present application, the scratch detection module in the stage training process includes a plurality of convolution operations Conv and activation functions, and the activation functions have two types represented as,

[0033] α is a learnable parameter

[0034]

[0035] Wherein, α is a learnable parameter greater than 0, and the initial definition value is 0.01; x of the two activation functions generally refers to the input value of the function, which refers to the feature matrix after the input image is abstracted; the scratch detection module has a U-shaped pyramid structure, that is, each layer includes a convolution operation and an activation function, the number of layers is generally not less than six, the operation transition between the front half of the layers uses 2 times down sampling, and the operation transition of the rear half uses 2 times up sampling, and the whole expression is U(x), the module input is the training data pair <x_input t , y_mask t >, which represents the to-be-processed data and the real scratch type feature, and the output is a group of scratch type features y_mask detectThe scratch detection module is optimized by the function optimization module to obtain the optimal parameters θ in the phased training process U .

[0036] In an embodiment of the present application, the optical flow transformation fusion module in the phased training process inputs a training data sequence pair <x_input t-i , x_input t , y t >, wherein x_input t-i represents the first i frames to be processed, x_input t represents the current frame to be processed, and y t represents the real reference mask. The optical flow transformation Flow needs to operate two frames respectively with the current frame two by two, first calculates the optical flow information between the two frames, then performs an affine transformation operation Warp, and finally outputs the complete transformed and fused sequence to the mask extraction module. For the superscript t, t-1 represents the previous frame, t+1 represents the next frame, and so on.

[0037] In an embodiment of the present application, the function optimization module is used to optimize the scratch detection module separately in the phased training process. The optimization target is the real scratch type y_mask t and the scratch type feature y_mask detect screened by the module. The minimum reconstruction loss is calculated between the two, so as to find the optimal parameters θ U . The formula is as follows,

[0038] L = ∑ (y_mask t -y_mask detect ) 2

[0039]

[0040] In the phased training process, the function optimization module is used to optimize the optical flow transformation fusion module and the mask extraction module together. The optimization target is the real reference mask y t and the extracted mask information y extract . The minimum reconstruction loss is calculated to obtain the optimal parameters θ F . The formula is as follows,

[0041] L = ∑ (y t -y extract ) 2

[0042]

[0043] In an embodiment of the present application, the scratch detection module, the optical flow transformation fusion module and the mask extraction module in the fusion optimization process use the optimal parameters θ obtained in the phased training process respectively U With θ F .

[0044] In the fusion optimization process, the multi-module fusion module is used to further extract features from the output results y_mask of the scratch detection module and the mask extraction module detect With y extract to enhance the feature information extraction capability.

[0045] First, the type feature information is integrated into the current frame information, then the mask information extracted by the mask extraction module is respectively subjected to mask operation with the input sequence, and finally the fusion operation is performed to output y_output fusion , which is given to the function optimization module for overall optimization.

[0046] The function optimization module in the fusion optimization process needs to optimize the parameters of the scratch detection module, the optical flow transformation fusion module, the mask extraction module and the multi-module fusion module as a whole, and the minimum reconstruction loss between the reference image gt and the fusion data y_output fusion is calculated to optimize the optimal overall parameter θ all , which is represented as,

[0047] L = ∑ (gt-y_output fusion ) 2

[0048]

[0049] In an embodiment of the present application, the parameters of the scratch detection module, the optical flow transformation fusion module, the mask extraction module and the multi-module fusion module in the inference process use the optimal parameters θ obtained in the fusion optimization process all . In the inference process, the data inflow module is used to convert the video or image sequence to be processed into a data form suitable for the model {frame t}, which represents all frames. The number of input frames can be increased or decreased as needed. In this method, five frames are taken as an example, so the current frame and the previous two frames and the next two frames, i.e. (frame t-2 , frame t-1 , frame t , frame t+1 , frame t+2 > are input, and the data is transmitted to the scratch detection module and the optical flow transformation fusion module after processing.

[0050] The scratch detection module in the inference process only inputs the current frame <frame t>, through the detection module U, get frame detect , the whole expression is,

[0051] frame detect = U(<frame t >)

[0052] Output frame detect is a set of scratch type features screened out, used for multi-module fusion module to further extract information and increase repair effect.

[0053] The input sequence of the optical flow transformation fusion module in the reasoning process is <frame t-2 , frame t-1 , frame t , frame t+1 , frame t+2 >, optical flow transformation Flow needs to operate two by two with the current frame respectively, so five frame input needs to operate four times, taking <frame t-1 , frame t > as an example, first calculate the optical flow information F t-1→t = Flow(frame t-1 , frame t ), then do affine transformation operation Warp, get the transformed feature W t-1→t , the calculation formula is,

[0054] F t-1→t = Flow(frame t-1 , frame t )

[0055] W t-1→t = Wrap(F t-1→t )

[0056] The final output is the complete transformation fusion sequence <w>to a mask extraction module,

[0057] <w>= <Warp(Flow(frame t` , frame t ))>, t

[0058] = (t-2, t-1, t+1, t+2)

[0059] During the inference process, the output sequence of the optical flow transformation fusion module <w>The four frames before and after the current frame are fused by optical flow transformation, and the mask extraction module MaskExtract needs to extract four groups of mask information, and the formula is,

[0060] <mask1, mask2, mask3, mask4> = MaskExtraCt( <w>)

[0061] Finally, the input is further processed in the multi-module fusion module.

[0062] In the reasoning process, the multi-module fusion module is used to integrate the output results of the scratch detection module and the mask extraction module, and further processing is performed, which can enhance the feature information extraction capability.

[0063] First, the current frame frame t and the scratch type feature information frame detect output by the scratch detection module are combined, and the type feature information is integrated into the current frame information, then the mask information extracted by the mask extraction module is respectively subjected to mask operation with the input sequence, and finally fusion operation Fuison is performed, and y fusion is output, and the overall formula is expressed as,

[0064] fusion t =frame t *frame detect *(1-<mask1,mask2,mask3,mask4>)

[0065] fusion all =frame t-2 *mask1+frame t-1 *mask2+frame t+1 *mask3+frame t+2 *mask4

[0066] y fusion =Fusion(fusion all ,fusion t )

[0067] In the reasoning process, the data flow module is used to convert the output frame y fusion processed by the multi-module fusion into an actual output video sequence.

[0068] By applying the method of the embodiment of the application, the sub-modules for repairing different types of defects are obtained through phased training, and then feature fusion optimization is performed, so that more accurate parameters are obtained, and the purpose of repairing the video is achieved. The technical scheme of the application trains different modules in stages, so that the model is highly modularized and more robust. The sequence processed by the model does not have the artifact problem caused by large motion, and the repair quality is improved. In a specific application example, Figure 2 The left is the real image with scratches, and the right is the processed effect diagram. The large irregular scratches on the complex background can be eliminated relatively cleanly. In another specific application example, Figure 3 The left is the real image with defects, the right is the processed effect diagram, large area stains and irregular defects can be removed.

[0069] It is to be understood that the example embodiments described herein are illustrative rather than limiting. Although one or more embodiments of the present application have been described in detail, various modifications can be made without departing from the spirit and scope of the present application, which is defined by the following claims.< / w> < / w> < / w> < / w>

Claims

1. A method for video inpainting based on multi-module hierarchical fusion learning, characterized in that, The training process includes a staged training process, a fusion optimization process, and an inference process. The staged training process is used for preliminary screening of a part of scratches by using a scratch detection module, input of training data to be processed <x_input t ,y_mask t > represents the data to be processed and the real scratch type feature, output of a set of screened scratch type features y_mask detect , optimization of the best parameters θ U by a function optimization module; the optical flow transformation fusion module inputs a sequence of training data pairs <x_input t-i ,x_input t ,x_input t+i ,y t >, wherein x_input t-i represents the first i frames to be processed, x_input t represents the current frame to be processed, x_input t+i represents the last i frames to be processed, y t represents a real reference mask, and optical flow transformation Flow needs to operate two by two between the front and rear frames and the current frame, first to calculate the optical flow information between the two frames, then to do affine transformation operation Warp, and finally to output the complete transformed fused sequence to the mask extraction module; The mask extraction module performs spatial alignment on the output sequence of the optical flow transformation fusion module, fuses the time sequence information of the previous frame and the current frame, and obtains accurate mask information y extract ; The function optimization module is used for optimizing the scratch detection module alone, and an optimization target is a real scratch type y_mask t and the screened scratch type feature y_mask detect , and the minimum reconstruction loss is calculated between the two, so as to find the optimal parameter θ U The multi-module fusion module is used for splicing the results output by the scratch detection module and the mask extraction module, fusing the results with a current processing frame and surrounding frames, and finally outputting a processing result to the function optimization module for parameter updating. The function optimization module optimizes the optical flow transformation fusion module and the mask extraction module together, and the optimization target is a real reference mask y t and the extracted mask information y extract , calculate the minimum reconstruction loss to obtain the optimal parameter θ F ; The fusion optimization process uses the best parameters θ obtained in the stage-by-stage training process for the scratch detection module, the optical flow transformation fusion module, and the mask extraction module U With θ F The function optimization module optimizes the parameters of the scratch detection module, the optical flow transformation fusion module, the mask extraction module, and the multi-module fusion module as a whole, optimizes the best overall parameters θ by calculating the minimum reconstruction loss between the reference image gt and the fusion data y_output fusion all ;​ The inference process uses the best parameters θ obtained in the fusion optimization process for the parameters of the scratch detection module, the optical flow transformation fusion module, the mask extraction module, and the multi-module fusion module all , converts the video or image sequence to be processed into a data form suitable for the model{frame t}, combines the current frame frame t with the scratch type feature information frame detect output by the scratch detection module, integrates the type feature information into the current frame information, then performs mask operation on the mask information extracted by the mask extraction module with the input sequence respectively, and finally performs fusion operation Fuison to output y fusion , the processed output frame y fusion is converted into the actual output video sequence.

2. The multi-module stepwise fusion learning based video inpainting method of claim 1, wherein, The scratch detection module includes a plurality of convolution operations Conv and activation functions, and the activation functions have two types, which are represented as: The scratch detection module structure is a U-shaped pyramid structure, that is, each layer includes a convolution operation and an activation function, the number of layers is not less than six, the transition between the front half of the layers uses a 2 times down-sampling operation, the transition between the back half of the layers uses a 2 times up-sampling operation, and the whole expression is U(x), the input of the scratch detection module is a pair of training data to be processed <x_input t , y_mask t > represents the data to be processed and the real scratch type feature, the output is a group of screened scratch type features y_mask detect , and the scratch detection module is optimized by a function optimization module to obtain optimal parameters θ U .

3. The multi-module stepwise fusion learning based video inpainting method of claim 1, wherein, The function optimization module is used for optimizing the scratch detection module alone, and an optimization target is a real scratch type y_mask t and the screened scratch type feature y_mask detect The minimum reconstruction loss is calculated between the two, so as to find the optimal parameter θ U The formula is represented as:

4. The multi-module stepwise fusion learning based video inpainting method of claim 1, wherein, The function optimization module optimizes the optical flow transform fusion module and the mask extraction module together, and an optimization target is a real reference mask y t and the extracted mask information y extract , and the minimum reconstruction loss is calculated to obtain the optimal parameter θ F The formula is represented as:

5. The multi-module stepwise fusion learning based video inpainting method of claim 1, wherein, Data flow in module is used in the inference process to convert the video or image sequence to be processed into a data form suitable for the model t} represents all frames, where the number of input frames is increased or decreased as needed, and after processing, the data is transmitted to the scratch detection module and the optical flow transformation fusion module.

Images