Picture quality improvement method based on steering feature separation

Abstract

The application discloses a picture quality improving method based on a guided feature separation, which comprises a training process and an inference process, wherein a feature rough extraction module is used to extract features after fusing input foreground and background data, and the features are given to a guided feature separation module; in the training process, parameters need to be transmitted to a function optimization module; the guided feature separation module uses a guided filtering technology to quickly identify and separate foreground and background features, and the separated features are given to a mask synthesis module; in the training process, parameters need to be transmitted to the function optimization module; the mask synthesis module is responsible for fusing the separated features and outputting the features to a data outflow module; in the training process, parameters need to be transmitted to the function optimization module; the function optimization module firstly performs stage training on parameters of the feature rough extraction module, and then performs optimization on the feature rough extraction, the guided feature separation and the mask synthesis module together, and finally all optimized modules are used in the inference process.

Description

Technical Field

[0001] This invention belongs to the field of video processing technology, specifically relating to a method for improving image quality based on guided feature separation. Background Technology

[0002] During video transmission, various compression techniques are used to reduce video size and transmission bandwidth, resulting in poor video quality, loss of details, and a reduced viewing experience for the receiving end.

[0003] Therefore, a method that can reduce video bitrate and transmission bandwidth while maintaining or restoring video quality is essential. One existing method is background restoration, which provides a set of clear background images of the same scene, analyzes and extracts the feature differences between the foreground and background, and then fuses the separated feature information. This method can ensure the accuracy of foreground information and significantly improve background quality. However, this method requires a high degree of foreground-background separation capability and a high ability to identify and suppress interference noise in the background image. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention provides an image quality improvement method based on guided feature separation, which is used to improve the algorithm's fault tolerance to the reference background and improve the image restoration quality.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] An image quality improvement method based on guided feature separation, including training and inference processes.

[0007] During training, a coarse feature extraction module is used as the input training sequence. <data x , data ref >, where data ref The input data to be trained x Corresponding reference background data; the feature coarse extraction module will <data x , data ref Downsampling yields the guide image pair The multiplier is arbitrary, and during training it is set to a random value between 1 and 4. Then... <data x , data ref Perform feature rearrangement to obtain lower-sized feature maps, and then perform feature extraction. The formula is as follows:

[0008]

[0009] The foreground and background mixed feature map is the coarsely extracted foreground and background mixed feature map, which is sent to the guided feature separation module together with the guide image pair The foreground and background mixed feature map is the coarsely extracted foreground and background mixed feature map, which is sent to the guided feature separation module together with the guide image pair The parameters of the feature coarse extraction module are separately optimized by the function optimization module;

[0010] The foreground and background mixed feature map is the coarsely extracted foreground and background mixed feature map, which is sent to the guided feature separation module together with the guide image pair The foreground and background mixed feature map is the coarsely extracted foreground and background mixed feature map, which is sent to the guided feature separation module together with the guide image pair The foreground and background mixed feature map is the coarsely extracted foreground and background mixed feature map, which is sent to the guided feature separation module together with the guide image pair

[0011] The foreground and background mixed feature map is the coarsely extracted foreground and background mixed feature map, which is sent to the guided feature separation module together with the guide image pair The foreground and background mixed feature map is the coarsely extracted foreground and background mixed feature map, which is sent to the guided feature separation module together with the guide image pair x , data ref > are synthesized to obtain the to-be-optimized frame in the training process The sequence is sent to the function optimization module;

[0012] The parameters of the feature coarse extraction module are separately optimized by the function optimization module according to to obtain the optimal parameters θ extract After that, the guided feature separation module and the mask synthesis module are jointly optimized, and the optimal overall parameters θ all are optimized by calculating the minimum reconstruction loss between the benchmark image GT and the to-be-optimized sequence ;

[0013] In the reasoning process, the feature coarse extraction module, the guided feature separation module and the mask synthesis module use the optimal overall parameters θ all obtained in the training process;

[0014] In the reasoning process, the data inflow module is used to convert the to-be-processed video or image sequence into a data form suitable for the model <frame x , frame ref >, wherein frame x is the to-be-processed video sequence and frame ref is the reference background frame under the corresponding scene, and the data is transmitted to the feature coarse extraction module after processing;

[0015] In the reasoning process, the feature coarse extraction module is used to downsample <frame x , frame ref > to obtain the guide image pair<img h , img l>, then the <frame x , ftame ref > is respectively subjected to feature rearrangement and feature extraction extract operation to obtain the foreground and background mixed feature map y feat , and the <img h , img l > is jointly processed by the guided feature separation module.

[0016] In the inference process, the guided feature separation module is used to separate the foreground and background mixed feature map y feat into a foreground and background region separated feature combination <fgr, bgr, pha>, and then the guided filtering operation is performed on the feature combination to obtain the corresponding high-resolution feature combination

[0017] In the inference process, the mask synthesis module is used to synthesize the and the input data <frame x , frame ref > to obtain the output frame Y after quality restoration.

[0018] In the inference process, the data outflow module is used to convert the output frame Y processed by the mask synthesis module into the actual output video sequence.

[0019] In a possible implementation, the guided feature separation module in the training process further includes a guided convolution, a mean filtering convolution, and a point-by-point convolution. The guided convolution first obtains the guided feature G l , and the <img > is subjected to the mean filtering convolution to obtain the relevant mean variance under low resolution, and then the linear transformation coefficient <A, b> is obtained by point-by-point convolution of the calculated relevant mean variance parameters, and the operation is as follows,

[0020]

[0021]

[0022] mean x =boxfilter(G l )

[0023]

[0024] var x =boxfilter(G l *G l )-mean x *mean x ​

[0025]

[0026] A=pointwise(<var x ,cov xy >)

[0027] b=mean y -A*mean x

[0028] Finally, the linear transform coefficients under low resolution are up-sampled as the guide feature G of the transform coefficients of the guide feature G h , and the guide feature G corresponding high-resolution features F

[0029] In a possible implementation, in the training process, the mask synthesis module synthesizes the guide feature G output by the guide feature separation module with the input data <data x , data ref > to obtain the to-be-optimized frame Y of the training process, and the operation is as follows:

[0030]

[0031] In a possible implementation, in the training process, the function optimization module optimizes the optimal overall parameter θ all , and the overall parameter is represented as,

[0032]

[0033]

[0034]

[0035]

[0036] In a possible implementation, in the inference process, the mask synthesis module synthesizes the guide feature G with the input data <frame x , frame ref > to obtain the final quality restored output frame Y, and the operation is as follows:

[0037]

[0038] The present invention has the following beneficial effects: The technical solution of the present invention proposes to use a guided feature separation technology to enhance the algorithm's ability to distinguish between foreground and background regions, improve the recognition of interference information in the background, and improve the image restoration quality even when the reference background has a lot of noise, thus ensuring the robustness of the algorithm and obtaining more accurate image quality. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating the steps of the image quality improvement method based on guided feature separation according to an embodiment of the present invention.

[0040] Figure 2 This is a schematic diagram illustrating the workflow of the guided feature separation module in the image quality improvement method based on guided feature separation according to an embodiment of the present invention.

[0041] Figure 3 This is a comparative diagram showing the input image and the image after quality improvement using this method in a specific application example of the present invention.

[0042] Figure 4 In another specific application example of this invention, a low-quality image is input;

[0043] Figure 5 In another specific application example of the present invention, and Figure 4 Corresponding reference background image;

[0044] Figure 6 In another specific application example of the present invention, and Figure 4 The corresponding effect image after processing by the existing background restoration method;

[0045] Figure 7 In another specific application example of the present invention, and Figure 4 The corresponding result image after processing with this method. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] Reference Figure 1 The diagram shows a flowchart of an image quality improvement method based on guided feature separation according to an embodiment of the present invention, including a training process and an inference process.

[0048] During training, a coarse feature extraction module is used as the input training sequence. <data x , dataref where data ref is the input data data x to be trained; the corresponding reference background data; the feature coarse extraction module down-samples <data x , data ref > to obtain the guide image pair The multiple is arbitrary, and is set to a random value between 1 and 4 during training. Then, <data x , datare f > is respectively subjected to feature rearrangement to obtain a feature map with a lower size, and then subjected to feature extraction extract operation, which is expressed by the formula:

[0049]

[0050] The foreground-background mixed feature map coarsely extracted is combined with the guide image pair to be processed by the guided feature separation module, and is transmitted to the function optimization module to optimize the parameters of the feature coarse extraction module separately.

[0051] During the training process, the guided feature separation module decomposes the foreground-background mixed feature map into a feature combination separated in the foreground and background regions Then, the guided filtering operation is performed on the feature combination to calculate the corresponding high-resolution feature combination

[0052] During the training process, the mask synthesis module synthesizes the output of the guided feature separation module with the input data <data x , data ref > to obtain the frame to be optimized in the training process The sequence is input into the function optimization module.

[0053] During the training process, the function optimization module optimizes the parameters of the feature coarse extraction module according to to obtain the optimal parameters θ extract After that, the guided feature separation module and the mask synthesis module are jointly optimized to obtain the optimal overall parameters θ all by calculating the minimum reconstruction loss between the benchmark image GT and the sequence to be optimized .

[0054] During the inference process, the feature coarse extraction module, the guided feature separation module and the mask synthesis module use the optimal overall parameters θ obtained during the training process.all ;

[0055] In the inference process, the data inflow module is used to convert the video or image sequence to be processed into a data form suitable for the model <frame x , frame ref >, where frame x is the video sequence to be processed and frame ref is the reference background frame under the corresponding scene, and the data is transmitted to the feature rough extraction module after processing;

[0056] In the inference process, the feature rough extraction module is used to downsample <frame x , frame ref > to obtain the guide image pair<img h , img l >, then perform feature rearrangement and feature extraction extract operations on <frame x , frame ref > respectively to obtain the foreground-background mixed feature map y feat , which is sent to the guided feature separation module together with the guide image pair<img h , img l >;

[0057] In the inference process, the guided feature separation module is used to first decompose the foreground-background mixed feature map y feat into a foreground and background region separated feature combination<fgr, bgr, pha>, then perform guided filtering operation on the feature combination to calculate the corresponding high-resolution feature combination

[0058] In the inference process, the mask synthesis module is used to synthesize with the input data <frame x , frame ref > to obtain the output frame Y after quality restoration;

[0059] In the inference process, the data outflow module is used to convert the output frame Y processed by the mask synthesis module into the actual output video sequence.

[0060] Through the above setting of the picture quality improvement method based on the guided feature separation, the data inflow module unifies the input data of various format types to obtain a format recognizable by the module; the data outflow module converts the data output by the mask synthesis module into a format consistent with the input data of the data inflow module; the feature coarse extraction module extracts features after fusing the input foreground and background data, and delivers parameters to the function optimization module in the training process; the guided feature separation module quickly identifies and separates foreground and background features by using guided filtering technology, and delivers the separated features to the mask synthesis module, which needs to deliver parameters to the function optimization module in the training process; the mask synthesis module is responsible for fusing and outputting the separated features to the data outflow module, which needs to deliver parameters to the function optimization module in the training process; the function optimization module first trains the parameters of the feature coarse extraction module in stages, and then optimizes the feature coarse extraction, guided feature separation and mask synthesis modules together, and finally all the optimized modules are used in the reasoning process.

[0061] In an embodiment of the present application, the guided feature separation module in the training process further includes guided convolution, mean filtering convolution and point-by-point convolution. The guided feature G l is obtained by the guided convolution The relevant mean variance under low resolution is obtained by mean filtering convolution calculation together with

[0062]

[0063]

[0064] mean x =boxfilter(G l )

[0065]

[0066] var x =boxfilter(G l *G l )-mean x *mean x

[0067]

[0068] A = pointwise(<var x , cov xy >)

[0069] b = mean y -A * mean x

[0070] Finally, the linear transform coefficients under low resolution are up-sampled as the guide feature G of the transform coefficients of the guide feature G h , and the guide corresponding high-resolution features

[0071] The above guide feature separation module uses the idea of guide filtering, obtains linear transform coefficients through low-resolution input, and then calculates high-resolution results through linear transformation. A series of linear transform coefficient extraction modules are composed of learnable neural networks, including guide convolution guide, box filter convolution box filter and pointwise convolution pointwise, to enhance the expression ability of linear transform coefficients. At the same time, such design can ensure that the features learned by the module are less, the calculation amount is more saved, and the separation effect is better.

[0072] In an embodiment of the present application, in the training process, the mask synthesis module synthesizes the output of the guide feature separation module with the input data <data x , data ref > to obtain the to-be-optimized frame of the training process. The operation is as follows:

[0073]

[0074] In an embodiment of the present application, in the training process, the function optimization module optimizes the best overall parameter θ all , which is represented as

[0075]

[0076]

[0077]

[0078]

[0079] In an embodiment of the present application, in the inference process, the mask synthesis module synthesizes with the input data <frame x , frame refThe operation of synthesizing to obtain the output frame Y after final quality restoration is as follows:

[0080]

[0081] As shown in Figure 3 is a specific application example, the left is the input picture, and the right is the result after quality improvement by the method. It can be found that the stripe noise existing on the input picture is removed, and the picture is cleaner; Figures 4 to 7 are, in turn, the input low-quality picture, the reference background picture, the effect picture of the existing background restoration method, and the effect picture of the method. It can be found that when the corresponding regions of the reference background picture and the input picture are the same (or are both backgrounds), the restoration effects of the two methods are similar, and both achieve the best effect. However, for the case where the corresponding regions of the reference background picture and the input picture are different, the method can obtain accurate and obviously improved results.

[0082] It should be understood that the example embodiments described herein are illustrative and non-limiting. Although one or more embodiments of the application are described in conjunction with the attached figures, it should be understood that various changes, modifications, and substitutions can be made by one of ordinary skill in the art without departing from the spirit and scope of the application as defined by the following claims.

Claims

1. A picture quality enhancement method based on steering feature separation, characterized by, The training process and the inference process are included, The feature rough extraction module is used to input training sequence <data x ,data ref > during the training process, wherein data ref is the input data data x to be trained and corresponding reference background data; the feature rough extraction module down-samples <data x ,data ref > to obtain a guide image pair , wherein the multiple is arbitrary, and is set to a random value between 1 and 4 during the training; then, the feature rearrangement is performed on <data x ,data ref > respectively, the feature map with a lower size is obtained, and the feature extraction extract operation is performed, and the formula is represented as: i.e. the foreground-background mixed feature map coarsely extracted, is sent to the guided feature separation module together with the guide image pair Meanwhile, the parameters of the feature coarsely extraction module are sent to the function optimization module for separate optimization. Meanwhile, the parameters of the feature coarsely extraction module are sent to the function optimization module for separate optimization. During training, a guided feature separation module is used to combine foreground and background feature maps. Decomposed into feature combinations that separate foreground and background regions Next, a guided filtering operation is performed on the feature combination to calculate the corresponding high-resolution feature combination. The mask synthesis module separates the guided feature from the output of the guided feature separation module in the training process The input data <data x , data ref > is synthesized to obtain a to-be-optimized frame of the training process The to-be-optimized frame is sent to the function optimization module The function optimization module in the training process is configured to Optimize the parameters of the feature extraction module alone to obtain the optimal parameters θ extract Then, the guiding feature separation module and the mask synthesis module are jointly optimized, and the optimal overall parameters θ are obtained by calculating the minimum reconstruction loss between the reference image GT and the frame to be optimized all ; The feature rough extraction module, the guide type feature separation module and the mask synthesis module in the reasoning process use the best overall parameter θ obtained in the training process all ; In the reasoning process, the data inflow module is used to convert each frame in the video or image sequence to be processed into a data form suitable for the model <frame x ,frame ref >, wherein frame x is the video sequence to be processed and frame ref is the reference background frame under the corresponding scene, and the data is transmitted to the feature rough extraction module after processing. The feature rough extraction module is used in the reasoning process to down-sample <frame x ,frame ref > to obtain a guide image pair<img h ,img l >, then the <frame x ,frame ref > is respectively subjected to feature rearrangement and feature extraction extract operation to obtain a foreground and background mixed feature map y feat , which is together with the guide image pair<img h ,img l > to be processed by the guide type feature separation module; The guided feature separation module is used in the reasoning process to separate the foreground and background mixed feature map y feat First, the foreground and background region separation feature combination <fgr, bgr, pha> is decomposed, and then a guided filtering operation is performed on the feature combination to calculate the corresponding high-resolution feature combination In the reasoning process, the mask synthesis module is used to synthesize the mask with the input data <frame ,frame x ,frame ref > to obtain the output frame Y after final quality restoration. In the inference process, the data flow-out module is used to convert the output frame Y processed by the mask synthesis module into the actual output video sequence. Wherein the guided feature separation module in the training process further comprises a guided convolution, a mean filtering convolution and a point-by-point convolution, the guided convolution first obtains the guided feature G l , which is subjected to the mean filtering convolution together with to obtain the relevant mean variance at low resolution, and then the relevant mean variance parameters obtained by calculation are subjected to the point-by-point convolution to obtain the linear transformation coefficient <A, b>, which is operated as follows, mean x = boxfilter(G l ) var x = boxfilter(G l *G l )-mean x *mean x A = pointwise(<var x ,cov xy >) b = mean - -A*mean x Finally, the linear transform coefficients at low resolution are upsampled as the guided feature G ％ transform coefficients, guiding to generate the corresponding high-resolution feature 2. The guide-based feature separation based picture quality enhancement method of claim 1, wherein, The mask synthesis module is used in the training process to separate the guided feature from the output of the guided feature separation module The input data <data x , data ref > is subjected to a synthesis operation to obtain the to-be-optimized frame of the training process The operation is as follows:

3. The directionally oriented feature-based picture quality enhancement method of claim 1, wherein, The function optimization module optimizes the best overall parameter θ during the training process all , which is represented as, 4. The directionally oriented feature-based picture quality enhancement method of claim 1, wherein, In the reasoning process, the mask synthesis module is used to synthesize the mask and the input data <frame to obtain the output frame Y after the final quality restoration. x ,frame ref The operation is as follows:

Images