Non-parametric local adaptive motion scene video super-resolution reconstruction method

Abstract

The invention relates to a super-resolution reconstruction method for a non-parametric local self-adaptive super-resolution video, belongs to the technical field of digital image processing, and is particularly suitable for reconstructing a high-resolution image from a low-resolution video sequence in a complex dynamic motion scene. According to the method, a non-parametric local self-adaptive super-division reconstruction model of a video time-space domain is established through global motion estimation in combination with a local gray difference loss function and a BTV regularization item. By introducing a weight matrix and an adaptive threshold parameter, the influence of a motion estimation error on a reconstruction result is reduced, and a gradient descent method is adopted to carry out iterative optimization. According to the method, a high-resolution image can be effectively reconstructed in a complex motion scene, the reconstruction precision of local details is improved, and the robustness to motion errors is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and in particular to a super-resolution reconstruction method of a motion scene video image based on local area adaptive adjustment, which is used for reconstructing a high-resolution image from a low-resolution video sequence. Background Art

[0002] Existing super-resolution reconstruction methods are mostly based on static images or simple low-resolution video sequences, ignoring the image blur problem caused by complex motion estimation errors in moving scenes. Especially in the fields of satellite video and security monitoring, due to shooting angles, light changes, and rapid movement of targets, traditional super-resolution methods are difficult to effectively improve image quality. To this end, it is necessary to further consider the dynamic changes of local areas and the impact of motion estimation errors on reconstruction results on the basis of motion estimation, so as to achieve more accurate high-resolution image reconstruction.

[0003] Existing super-resolution reconstruction methods can be mainly divided into interpolation-based, frequency domain conversion-based and deep learning-based methods. Although the interpolation method is simple, it can easily lead to blurred image edges and loss of details when processing complex motion scenes [1] (Zhang, H., et al. "Image Super-Resolution Using Iterative Back Projection." IEEE Transactions on Image Processing, vol. 6, no. 5, 2015, pp. 1239–1243.); the frequency domain conversion method has certain advantages in processing high-frequency information, but it is very sensitive to motion estimation errors and cannot effectively deal with nonlinear deformation problems in complex scenes [2] (Farsiu, S., et al. "Fast and Robust Multiframe SuperResolution." IEEE Transactions on Image Processing, vol. 13, no. 10, 2004, pp. 1327–1344.); although the deep learning-based method has achieved good results in super-resolution reconstruction of static images, it is difficult to apply it to real-time image reconstruction in dynamic scenes in practical applications because it relies on a large amount of training data and has poor robustness to motion estimation [3] (Dong, C., et al. "Image Super-Resolution Using DeepConvolutional Networks." IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 2, 2016, pp. 295–307.).

[0004] Based on the above methods, it can be seen that under the premise of motion estimation errors, general reconstruction methods currently have "ringing" or "tailing" phenomena, that is, the processing effect on dynamic low-resolution images is poor. In order to improve the robustness of the model, avoid machine errors caused by training data sets, and improve local reconstruction accuracy, there is an urgent need for a reconstruction method that can effectively handle local detail motion estimation errors and consider dynamic scenes. Summary of the invention

[0005] The present invention proposes a method that can effectively cope with motion estimation errors in complex dynamic motion scenes, improve the reconstruction accuracy of local details, and achieve efficient super-resolution image reconstruction through an adaptive adjustment mechanism.

[0006] To achieve the above object, the present invention proposes a non-parametric local adaptive motion scene video super-resolution reconstruction method, comprising the following steps:

[0007] S1: Extract frames from the input low-resolution video sequence, use the global motion estimation method to estimate the motion matrix between adjacent resolution image frames, and then calculate the motion vector of each frame relative to the reference frame through the motion vectors between adjacent frames;

[0008] S2: The scope of the motion parameter is defined as the grayscale difference constraint of the image before and after reconstruction in an arbitrary local window. Then the loss function term of the non-parametric local adaptive algorithm for the sequence low-resolution image t∈[1,…,T] is defined as follows:

[0009]

[0010] Among them, (k, l) represents the pixel point, Ω represents the global pixel point, t represents the frame number, T represents the total number of frames in the low-resolution video sequence, (i, j) represents the point in the local neighborhood, w[k, l, i, j, t] represents the weight definition function considering three dimensions, and the matrices D and B are the downsampling matrix and blur matrix respectively. Indicates the image area of ​​size (p*p) extracted from the high-resolution image with the image point (k,l) as the center. represents the image area of ​​size (q*q) extracted from the low-resolution image with the image point (i, j) as the center, x represents the high-resolution image, y t The low-resolution image corresponding to this frame number;

[0011] S3: Select the BTV constraint as a regular term to maintain the image edge features. The expression formula is:

[0012]

[0013] Among them, p represents the maximum translation, α represents the scale weight factor, and its range is [0,1]. Respectively represent a horizontal translation of l pixels and a vertical translation of m pixels;

[0014] S4: Combining the loss function terms of steps S2 and S3 and the BTV regularization term, the grayscale difference between pixels is used to affect the solution weight definition, and the MAP regularization method is used to establish the model:

[0015]

[0016] Among them, λ is the regularization parameter, F(x) represents the guarantee item, and Z(x) represents the regularization item;

[0017] S5: Based on steps S2, S3 and S4, a super-resolution reconstruction model of a non-parametric local adaptive algorithm in the spatiotemporal domain of the video is constructed;

[0018] S6: Solve the objective function using the gradient descent method, set the maximum number of iterations N, and continue to loop if the number of iterations has not been reached. The iteration formula is as follows:

[0019]

[0020] where x n+1 represents the high-resolution image after iterative update, x n is the initial term of iterative update, α is the learning rate, is the gradient of the objective function;

[0021] S7: When the maximum number of iterations N is reached, terminate and output the high-resolution image.

[0022] Preferably, for the super-resolution reconstruction objective function in step S4, the super-resolution reconstruction objective function is as follows:

[0023]

[0024] where (k, l) represents a pixel point, Ω represents all pixel points globally, t represents the frame number, T represents the total number of frames in the low-resolution video sequence, (i, j) are points within the local neighborhood range, w[k, l, i, j, t] represents the weight definition function considering three dimensions, matrices D and B are the downsampling matrix and the blurring matrix respectively, represents the image region of size (p * p) extracted centered on the pixel point (k, l) in the high-resolution image, and similarly represents the image region of size (q * q) extracted centered on the pixel point (i, j) in the low-resolution image.

[0025] Furthermore, the solution formula for the weight matrix w[k, l, i, j, t] in step S4:

[0026]

[0027] where each parameter in represents the local patch gray level difference described in the objective function. The smaller the difference, the higher the local similarity of the image; δ r is a parameter that controls the influence of the gray level difference between two pixels; the function f is expressed as a function of a geometric distance; the entire weight function adopts an exponential function form, indicating that the increase in gray level difference and geometric distance will rapidly reduce the weight. Thus, during the reconstruction process, more reliance is placed on pixels with similar gray levels and close spatial proximity.

[0028] Furthermore, when the number of iterations n in step S6 reaches the corresponding number N, the output includes: judging whether n < N. If yes, return to step S4; if no, enter step S7.

[0029] From the above, the non-parametric local adaptive motion scene video super-resolution reconstruction method proposed in the present invention uses a global motion estimation method to estimate the motion vector between adjacent frames; constructs a super-resolution reconstruction model based on BTV regularization and adaptive local area estimation, and controls the influence of motion estimation errors on the results by introducing a weight matrix; effectively constrains the noise in the high-resolution image reconstruction process by calculating the norm of the adaptive threshold parameter; and uses the gradient descent method to iteratively optimize the super-resolution reconstruction process to ensure that the final result is output after reaching a predetermined number of iterations. This method can adaptively reduce the contribution of low-resolution images with large motion estimation errors to the reconstruction process by introducing a robust estimation method, thereby enhancing the robustness to errors. Experiments have verified the effectiveness of this method in processing complex motion scenes, and achieved high-precision super-resolution reconstruction under the influence of errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is an overall flow chart of a non-parametric local adaptive motion scene video super-resolution reconstruction method of the present invention. DETAILED DESCRIPTION

[0031] The present invention proposes a non-parametric local adaptive motion scene video super-resolution reconstruction method, which is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] The present invention proposes a non-parametric local adaptive motion scene video super-resolution reconstruction method, which can realize the super-resolution reconstruction of satellite images. During the operation, the low-resolution image and the reconstructed high-resolution image difference are calculated by combining the global pixel points of the high-resolution image, the number of video frames, and the neighborhood pixel points of the low-resolution image. It effectively copes with the motion estimation error problem in the low-resolution video sequence and completes the super-resolution reconstruction of high-quality satellite images. The overall process of the method is as follows Figure 1 As shown, the following steps are included:

[0033] Step S1: Obtain satellite image sequences, extract inter-frame image data from low-resolution video sequences, and use global motion estimation technology to determine the motion vector between adjacent frames. The motion vector is defined as the direction and amplitude of inter-frame motion, and the optical flow method is usually used to describe the displacement of inter-frame pixels. The specific steps are as follows:

[0034] S1-1) Acquire satellite image sequences;

[0035] The present invention has the following requirements for satellite sequence images:

[0036] The image size is related to computer performance. Generally, the original size of the input image can be selected, such as 1024×1024. If the sequence image is a satellite video, at least 20 seconds of video is required (a total of 20 seconds * 25 frames). If the sequence image is not a video but a continuously shot picture, no less than 200 frames are required). The total number of satellite sequence images obtained is T frames.

[0037] S1-2) Perform global motion estimation on the selected satellite images. The motion estimation effect of static objects is better, but the motion of dynamic objects such as vehicles and aircraft will have "tailing" and other phenomena, and the estimation effect is poor;

[0038] S1-3) Combine the constraints of the "edge" of dynamic objects in motion segmentation and the grayscale information constraints in the spatial domain between frames to jointly estimate the motion field and realize motion estimation.

[0039] Step S2: To address the impact of motion estimation errors on reconstruction results, the similarity between local regions x and y in the time domain t of satellite video data is considered, and a loss function term is constructed. The scope of the motion parameter is defined as a grayscale difference constraint of the image before and after reconstruction in an arbitrary local window.

[0040] Then the loss function of the non-parametric local adaptive algorithm for the sequence of low-resolution images t∈[1,…,T] is defined as follows:

[0041]

[0042] The parameters are defined as follows:

[0043] (1) Ω represents the image area, that is, all pixel positions of the high-resolution image involved in the reconstruction process, and (k, l) represents the pixel coordinates on the high-resolution image;

[0044] (2) T represents the total number of frames of the low-resolution video. Each frame of the low-resolution image is used to reconstruct the high-resolution image, thereby improving the reconstruction accuracy from the time dimension.

[0045] (3) N represents the neighborhood range, (i, j) is the coordinate of the neighborhood pixel in the low-resolution image, which means that the reconstruction process not only considers a single pixel point, but also the relationship between neighborhood pixels;

[0046] (4) The w[k,l,i,j,t] weight function is used to measure the similarity between high-resolution pixels and low-resolution pixels to improve the accuracy of motion estimation;

[0047] (5) The matrices D and B are the downsampling matrix and blur matrix, respectively, which downsample the high-resolution image to simulate the low-resolution image.

[0048] (6) It is expressed as an operation of extracting a high-resolution patch of size (p*p) with the image point (k,l) as the center. It is expressed as an operation of extracting a low-resolution facet size (q*q) with the image point (i, j) as the center, and comparing the high-resolution image with the low-resolution image;

[0049] (7) x is the high-resolution image, y t The low-resolution image corresponding to this frame number;

[0050] (8) The introduction of L2 norm is used to calculate the error between the high-resolution simulated low-resolution image and the low-resolution observed image, thus improving the reconstruction accuracy;

[0051] This embodiment downsamples the high-resolution image according to the formula to simulate the low-resolution image, reducing the pixel dimension by half;

[0052] Step S3: Select BTV as a regularization term to maintain image edge features;

[0053] The expression formula is:

[0054]

[0055] The definition of each parameter and the setting of this embodiment are as follows:

[0056] (1) P represents the maximum range of translation, which is used to limit the range of translation operation to prevent the introduction of irrelevant information caused by excessive displacement. The translation range is 5 pixels.

[0057] (2) α is a scale weight factor, which is a coefficient that controls attenuation. Its function is to weight the difference after translation according to the increase in displacement distance. l and m represent the number of translation steps in the horizontal and vertical directions. The larger the number of translation steps, the smaller the impact of the difference.

[0058] (3) It represents the absolute value of the pixel difference of the high-resolution image x under different displacements. The L1 norm is used to preserve edge information.

[0059] The BTV regularization term of this embodiment uses the L1 norm as a fidelity term to ensure the robustness of the motion estimation error.

[0060] Step S4: Combining the loss function terms of steps S2 and S3 and the BTV regularization term, the grayscale difference between pixels is used to affect the solution weight definition, and the MAP regularization method is used to establish the model:

[0061]

[0062] Wherein, λ is a regularization parameter, and in this embodiment, the value is between 0.01 and 0.1, and the value is adjusted through multiple experiments.

[0063] Step S5: Based on S2, S3 and S4, a super-resolution reconstruction model of a non-parametric local adaptive algorithm in the spatiotemporal domain of the video is constructed;

[0064] The super-resolution reconstruction objective function is:

[0065]

[0066] The weight definition function is:

[0067]

[0068] Among them, δ r The parameter controls the influence of the gray level difference between two pixels, and the value range is between 5-20 according to the noise level of the image and the motion estimation error; the function F is expressed as a function of the geometric distance. The weight matrix is ​​adaptively adjusted to reduce the influence of the area with large motion estimation error on the reconstruction.

[0069] Step S6: Use the gradient descent method to iterate, and determine whether the number of iterations n reaches the maximum number of iterations N. If not, continue the cycle;

[0070] The iteration formula is:

[0071]

[0072] Among them, x n+1 is the high-resolution image after iterative update, X N is the initial term of iterative update, α is the learning rate, is the gradient of the objective function.

[0073] Step S7: When the maximum number N is reached, terminate and output a high-resolution image.

[0074] By adaptively adjusting the weights of local regions and combining BTV regularization and gradient descent optimization, the final high-resolution reconstructed image is generated. The image retains the details in complex dynamic scenes, especially the local regions with motion estimation errors are effectively processed.

[0075] The above description is only a specific implementation of the present invention, and the protection scope of the present invention is not limited thereto. Any person familiar with the technology can understand and think of any changes or substitutions within the technical scope disclosed by the present invention, which should be included in the scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A non-parametric local adaptive motion scene video super-resolution reconstruction method, characterized in that: The steps include: S1: Extract frames from the input low-resolution video sequence, use the global motion estimation method to estimate the motion matrix between adjacent resolution image frames, and then calculate the motion vector of each frame relative to the reference frame through the motion vectors between adjacent frames; S2: The scope of the motion parameter is defined as the grayscale difference constraint of the image before and after reconstruction in an arbitrary local window. Then the loss function term of the non-parametric local adaptive algorithm for the sequence low-resolution image t∈[1,…,T] is defined as follows: Among them, (k, l) represents the pixel point, Ω represents the global pixel point, t represents the frame number, T represents the total number of frames in the low-resolution video sequence, (i, j) represents the point in the local neighborhood, w[k, l, i, j, t] represents the weight definition function considering three dimensions, and the matrices D and B are the downsampling matrix and blur matrix respectively. Indicates the image area of ​​size (p*p) extracted from the high-resolution image with the image point (k,l) as the center. represents the image area of ​​size (q*q) extracted from the low-resolution image with the image point (i, j) as the center, x represents the high-resolution image, y t The low-resolution image corresponding to this frame number; S3: Select the BTV constraint as a regular term to maintain the image edge features. The expression formula is: Among them, p represents the maximum translation, α represents the scale weight factor, and its range is [0,1]. Respectively represent a horizontal translation of l pixels and a vertical translation of m pixels; S4: Combining the loss function terms of steps S2 and S3 and the BTV regularization term, the grayscale difference between pixels is used to affect the solution weight definition, and the MAP regularization method is used to establish the model: Among them, λ is the regularization parameter, F(x) represents the guarantee item, and Z(x) represents the regularization item; S5: Based on steps S2, S3 and S4, a super-resolution reconstruction model of a non-parametric local adaptive algorithm in the spatiotemporal domain of the video is constructed, and the super-resolution reconstruction objective function is: The weight definition function is: Among them, δ r The parameter controls the influence of the gray level difference between two pixels, and the function f is expressed as a function of geometric distance; S6: Use the gradient descent method to solve the objective function, set the maximum number of iterations N, and continue the loop if the number is not reached. The iteration formula is: Among them, x n+1 represents the high-resolution image after iterative update, x n is the initial term of iterative update, α is the learning rate, is the gradient of the objective function; S7: When the maximum number N is reached, terminate and output a high-resolution image.

Images