Motion blur suppression and detail restoration method for drone remote observation picture

Abstract

The present application relates to the field of image processing and computer vision, and discloses a motion blur suppression and detail recovery method for remote observation picture of unmanned aerial vehicle. The method is aimed at the motion blur caused by the change of flight attitude and vibration of unmanned aerial vehicle. Firstly, the degraded image containing motion blur is obtained, and the motion blur point spread function is adaptively estimated based on gradient saliency detection region and fusion of local structure anisotropy features. Then, a regularization deconvolution model with total variation and spatial weighted gradient constraint is constructed to obtain the preliminary restoration image by deconvolution. Finally, the preliminary restoration image is subjected to multi-scale frequency domain decomposition, and the final clear image is generated through cross-scale nonlinear fusion and detail reconstruction, without manual intervention, which can effectively suppress ringing artifacts and recover image details.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision technology, specifically to a method for motion blur suppression and detail restoration of images viewed remotely by unmanned aerial vehicles (UAVs). Background Technology

[0002] In remote observation applications of drones, due to factors such as changes in flight attitude, airflow disturbances, and mechanical vibrations, the video footage captured by image sensors is often accompanied by complex motion blur, resulting in a severe deterioration in image quality and a significant loss of detailed information.

[0003] Traditional image deblurring methods often assume that the blur kernel is uniform and known. However, motion blur in UAV scenarios is spatially variable, directional, and inconsistent. Conventional blind deconvolution algorithms are difficult to accurately estimate the blur kernel and are prone to producing ringing artifacts or oversmoothing.

[0004] Furthermore, existing technologies often fail to effectively distinguish between real edges and noise when processing low signal-to-noise ratio images, resulting in insufficient detail recovery or noise amplification.

[0005] Therefore, there is an urgent need for a robust method that can adaptively estimate the motion blur point spread function and preserve and enhance image details while suppressing blur. Summary of the Invention

[0006] To address the aforementioned technical shortcomings, the purpose of this invention is to provide a method for motion blur suppression and detail restoration of images viewed remotely by drones, thereby solving the problems of insufficient motion blur suppression and poor detail restoration in the prior art.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for motion blur suppression and detail restoration of images viewed remotely by a drone, comprising: Step S1: Acquire a degraded image containing motion blur from the drone during flight; Step S2: In the degraded image, based on gradient detection, determine the gradient saliency region, and fuse the anisotropic features of the local structure of the image to adaptively estimate the motion blur point spread function. Step S3: Based on the motion blur point spread function, construct a regularized deconvolution model with edge protection constraints, and perform deconvolution processing on the degraded image to obtain a preliminary restored image; wherein, the data fidelity term of the regularized deconvolution model is used to characterize the convolution relationship between the degraded image, the motion blur point spread function and the image to be restored, and its constraint term includes a total variation regularization sub-term and a spatially weighted gradient constraint sub-term; Step S4: Perform multi-scale frequency domain decomposition on the preliminary restored image to obtain multi-scale frequency domain sub-bands; then perform cross-scale nonlinear fusion and detail reconstruction on the multi-scale frequency domain sub-bands to generate the final restored image.

[0008] Preferably, in one possible implementation of the first aspect, step S1 includes: Receive the raw video stream transmitted by the image sensor of the drone while it is in flight; Based on the real-time motion data provided by the UAV flight control system, estimate the global motion trend between frames; By combining global motion trends, the pixel displacement vector field between adjacent frames in the original video stream is calculated; Based on the statistical characteristics of the pixel displacement vector field, degraded image frames containing significant motion blur are automatically selected from the original video stream.

[0009] Preferably, in one possible implementation of the first aspect, in step S2, determining the gradient saliency region specifically involves: Calculate the gradient magnitude map and gradient direction map of the degraded image; On the gradient magnitude map, the variance of the gradient magnitude in the neighborhood of each pixel is calculated using a sliding window method. On the gradient direction map, the angular consistency measure of the gradient direction is calculated using the same sliding window method. Pixel regions with gradient magnitude variance greater than the first threshold and gradient direction consistency higher than the second threshold are identified as gradient saliency regions.

[0010] Preferably, in one possible implementation of the first aspect, the anisotropic features of the local structure of the fused image specifically include: Within the gradient saliency region, calculate the structure tensor matrix of each image patch and solve for its principal eigenvalues, secondary eigenvalues, and principal eigenvector directions; Calculate the anisotropy coefficients based on the principal and secondary eigenvalues; Based on the direction of the principal eigenvector, the initial fuzzy direction estimate obtained from the gradient magnitude map and Radon transform is corrected.

[0011] Preferably, in one possible implementation of the first aspect, the adaptive estimation of the motion blur point diffusion function specifically comprises: Based on the corrected fuzzy direction and anisotropic coefficients, a motion fuzzy point diffusion function kernel model is constructed. By analyzing the energy diffusion width of the Radon transform projection curve at the corresponding corrected angle, the final fuzzy length estimate is obtained, thus completing the adaptive estimation of the motion fuzzy point diffusion function.

[0012] Preferably, in one possible implementation of the first aspect, in step S3, the regularized deconvolution model is expressed as a minimization problem of the following energy function:

[0013] in, For degraded images, Let be the motion blur point spread function. For the image to be restored, This represents the convolution operation. Denotes the total variation regularity term. Represents the gradient operator, The spatial variation weight matrix is ​​derived from the image content. This represents the predefined norm function. and This is the regularization parameter.

[0014] Preferably, in one possible implementation of the first aspect, the deconvolution processing of the degraded image specifically includes: By employing a variable splitting strategy and introducing auxiliary variables, the problem of minimizing the energy function is decomposed into subproblems concerning the restored image and each auxiliary variable. Within the framework of the alternating direction multiplier method, each subproblem is solved iteratively. The subproblems concerning the restored image are solved in the Fourier domain, while the subproblems involving regularization constraints are solved using the corresponding proximal operators.

[0015] Preferably, in one possible implementation of the first aspect, the deconvolution process includes a motion blur point spread function update step: After completing one or more iterations, fix the current restored image, and within the gradient saliency region, re-estimate the parameters of the motion blur point spread function based on the gradient information of the current restored image; The updated motion blur point diffusion function is used to calculate the data fidelity term in subsequent iterations.

[0016] Preferably, in one possible implementation of the first aspect, step S4 involves multi-scale frequency domain decomposition of the initially restored image, specifically: The initially restored image is decomposed into multiple scale pyramids to obtain multiple image layers of different scales. For each layer of the pyramid image, a multi-resolution transformation is performed to decompose it into low-frequency sub-bands and high-frequency sub-bands in multiple directions; Multi-resolution transforms include wavelet transform, contour wave transform, or shear wave transform.

[0017] Preferably, in one possible implementation of the first aspect, the cross-scale nonlinear fusion and detail reconstruction of multi-scale frequency domain subbands specifically includes: Calculate the correlation measure of the transform coefficients of the same spatial location in high-frequency subbands at different scales; Based on the correlation metric, an adaptive weight is constructed for fusing high-frequency subband coefficients at different scales; Using adaptive weights, the high-frequency subband coefficients of at least two different scales are fused; Based on the fused high-frequency subband and the low-frequency subband at the top of the pyramid, multi-resolution inverse transform and pyramid reconstruction are performed to obtain the final restored image.

[0018] The beneficial effects of this invention are as follows: by fusing gradient saliency detection with local structural anisotropy features, this invention achieves adaptive and accurate estimation of the diffusion function of motion fuzzy points, avoiding the bias caused by the reliance on global assumptions in traditional methods.

[0019] Based on this, a dual regularized deconvolution model with total variation and spatial weighted gradient constraints was constructed, which effectively suppressed the amplification of deconvolution noise while protecting the image edge structure and reducing ringing artifacts.

[0020] Furthermore, multi-scale frequency domain decomposition and cross-scale nonlinear fusion are performed on the initially restored image, which makes full use of the complementary information of high-frequency details at different scales, significantly improving the richness and naturalness of detail restoration.

[0021] This method requires no manual intervention and can automatically select degraded frames from the original video stream and complete the entire process. It is suitable for various complex flight environments, effectively improves the visual quality of UAV remote observation images, and provides a clearer image foundation for subsequent target recognition and tracking tasks. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This application provides a flowchart of a method for motion blur suppression and detail restoration of images viewed remotely by drones. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Example 1: As Figure 1 As shown, the present invention provides a method for motion blur suppression and detail restoration of images viewed remotely by unmanned aerial vehicles, comprising: Step S1: Acquire a degraded image containing motion blur from the drone in flight.

[0026] In this embodiment, the raw video stream is received via the UAV's onboard image sensor interface. The video stream has an image resolution of 1920×1080 pixels and a frame rate of 30 frames per second. Simultaneously with acquiring the video stream, real-time motion data transmitted by the UAV flight control system via a serial communication interface is received. This real-time motion data includes three-axis angular velocity and three-axis acceleration data provided by the inertial measurement unit, as well as latitude, longitude, and altitude data provided by the global positioning system. Based on this real-time motion data, a Kalman filter algorithm is used to estimate the global motion trend between adjacent video frames, resulting in a six-parameter affine transformation model describing the overall inter-frame motion.

[0027] Specifically, the initial estimate of the inter-frame pose change is obtained by inertial navigation calculation using inertial measurement unit data, and then corrected and fused using global positioning system data, finally outputting a rotation matrix representing the rotation. Translation vector representing translation .

[0028] Based on the estimated global motion trend, the following approach is adopted: The optical flow algorithm calculates the dense pixel displacement vector field between adjacent frames in the original video stream. Before calculation, the reference frame is first subjected to global motion compensation using an estimated affine transformation model to eliminate most of the pixel displacement caused by the drone's own flight motion, allowing the optical flow algorithm to focus more on the relative displacement caused by independent motion or high-frequency vibration within the scene. Optical flow algorithms approximate image intensity as a polynomial expansion and solve for the displacement vector of each pixel by analyzing the changes in polynomial coefficients across consecutive frames. In this embodiment, the number of pyramid layers for optical flow calculation is set to 4, the scaling factor for each layer is 0.5, the window size is 15×15 pixels, and the number of iterations is 3.

[0029] After obtaining the pixel displacement vector field, statistical feature analysis is performed on it to automatically filter degraded image frames containing significant motion blur. The average displacement amplitude of the entire frame's displacement vector field is calculated. Consistency measure with displacement direction Average displacement amplitude Calculated by the arithmetic mean of the magnitudes of all pixel displacement vectors. Orientation consistency measure. This is obtained by calculating the direction angle histogram of all displacement vectors, and taking the ratio of the number of vectors contained in the main peak of the histogram to the total number of vectors. A selection threshold is set, and when the corresponding frame... Greater than the preset amplitude threshold (For example, 5 pixels) and its Less than the preset directional consistency threshold When the value is 0.7 (e.g., if the image frame has significant spatial blur due to complex, non-uniform motion), it is marked as a degraded image frame to be processed. This selection process is automatic, and the final output is the selected degraded image frame for subsequent processing.

[0030] Step S2: In the degraded image, based on gradient detection, determine the gradient saliency region, and fuse the anisotropic features of the local structure of the image to adaptively estimate the motion blur point spread function.

[0031] In this embodiment, gradient detection is first performed to calculate the gradient magnitude map and gradient direction map of the degraded image. The Sobel operator is used to calculate the first derivatives of the image in the horizontal and vertical directions, and then the gradient magnitude and gradient direction at each pixel location are calculated. The gradient magnitude map reflects the intensity information of the image edges, while the gradient direction map reflects the normal direction of the edges.

[0032] Next, gradient saliency regions are determined. On the gradient magnitude map, a 15×15 pixel sliding window is used to traverse the image, and the variance of the gradient magnitude of all pixels in the neighborhood of the center pixel of each window is calculated. On the gradient direction map, a gradient direction consistency metric is calculated using a sliding window of the same size. This metric is obtained by calculating the ratio of the magnitude of the unit vector sum of the gradient direction angles of all pixels within the window to the total number of pixels within the window. Pixel regions with gradient magnitude variance greater than a first threshold and gradient direction consistency higher than a second threshold are identified as gradient saliency regions. In this embodiment, the first threshold is set to 500, and the second threshold is set to 0.8. Gradient saliency regions typically correspond to image portions with clear edge structures, and the gradient information from these regions is more reliable for estimating motion blur parameters.

[0033] Then, the anisotropic features of the local image structure are fused. Within the identified gradient saliency regions, image patches of size 7×7 pixels are extracted, centered on each pixel. For each image patch, its structure tensor matrix is ​​computed. (Structure tensor matrix...) The mathematical expression for this is obtained by taking the outer product of the horizontal and vertical gradients of all pixels within the image block and summing them:

[0034] in, and Representing pixels The first derivatives of the image in the horizontal and vertical directions are calculated. The eigenvalues ​​of the structure tensor matrix of each image patch are then determined through eigenvalue decomposition. Secondary eigenvalues and the direction of the principal eigenvector corresponding to the principal eigenvalue. The anisotropy coefficients of the region are calculated based on the principal and secondary eigenvalues. The calculation formula is:

[0035] in It is a very small positive number used to prevent division by zero errors. The anisotropy coefficient reflects the directional strength of the local image structure; the closer its value is to 1, the stronger the directional texture of the region.

[0036] Simultaneously, a Radon transform is performed based on the gradient magnitude map of the entire degraded image to obtain an initial blur direction estimate. The Radon transform projects the image along straight lines at different angles. In this embodiment, Radon projections at 180 angles are calculated in a step size of 1° within the range of 0° to 179°. The variance of the projection curve at each projection angle is calculated. Initial blur direction This corresponds to the angle that maximizes the variance of the projection curve, because motion blur typically leads to the highest concentration of projection energy along the blur direction. This is based on the direction of the principal eigenvector calculated within the gradient saliency region. For initial fuzzy direction estimation Correction is performed. The correction method is to calculate the direction of the average principal eigenvector in each gradient significance region. The difference, expressed as anisotropy coefficient As a weight, a weighted average is calculated based on the differences across all regions to obtain the orientation correction amount. Final corrected blur direction .

[0037] Finally, the motion blur point spread function is adaptively estimated based on the corrected blur direction. Construct a one-dimensional linear motion fuzzy point spread function kernel model The kernel model is a kernel of length [length missing]. , direction is The line segment function whose sum of all elements is 1. Fuzzy length. The estimation is achieved by analyzing the Radon transform projection curve at the angle. The energy diffusion width is obtained. Specifically, it is extracted from... The projection curve at the angle is calculated, and its autocorrelation function is used. The width corresponding to the point where the peak value of the autocorrelation function drops to half is taken as the estimate of the energy diffusion width. Then, after scaling, the blur length estimate in pixels is obtained. In this embodiment, the motion blur point diffusion function is completed through this process. Adaptive estimation of motion blur point spread function It is a size of A matrix whose non-zero elements are distributed in the direction of On the straight line.

[0038] Step S3: Based on the motion blur point spread function, construct a regularized deconvolution model with edge protection constraints, and perform deconvolution processing on the degraded image to obtain a preliminary restored image; wherein, the data fidelity term of the regularized deconvolution model is used to characterize the convolution relationship between the degraded image, the motion blur point spread function and the image to be restored, and its constraint term includes a total variation regularization sub-term and a spatially weighted gradient constraint sub-term.

[0039] In this embodiment, the regularized deconvolution model is expressed as a problem of minimizing the energy function of the image to be restored x, and its mathematical form is:

[0040] in, The input is a degraded image. Let x be the motion blur point spread function obtained through adaptive estimation, and let x be the preliminary restored image to be solved. The symbol is... This represents a two-dimensional convolution operation. (Data fidelity item) Used to constrain the restored image After fuzzy kernel The convolution should approximate the observed degraded image as closely as possible. . The total variation regularization term is defined as the sum of the gradient magnitudes in the horizontal and vertical directions of the image, i.e. ,in and These represent the difference operators in the horizontal and vertical directions, respectively. This regularization term effectively suppresses noise amplification and ringing artifacts that may be introduced during deconvolution by promoting the sparsity of the image gradient distribution. Represents the gradient operator, It is a spatial variation weight matrix derived from image content, where each element... The gradient magnitude is calculated based on the currently estimated gradient magnitude map of the preliminary restored image, specifically as follows: ,in The image estimate obtained from the previous iteration, In this embodiment, to adjust the parameters, The value is 5. The purpose of this weight matrix is ​​to assign larger weights to flat areas of the image to strengthen gradient constraints and smooth noise, and to assign smaller weights to edge areas to protect edge details from being over-smoothed. This represents a preset norm function; in this embodiment, the L1 norm is selected, i.e. . and This is a regularization parameter used to balance the weights between the data fidelity term and the two regularization constraint terms. In this embodiment, it is set to... , .

[0041] To solve this energy function minimization problem, a variable splitting strategy and an alternating direction multiplier method framework are employed. First, two auxiliary variables are introduced. and The original problem is equivalently transformed into the following constrained optimization problem:

[0042]

[0043] As constraints, then, by introducing Lagrange multipliers , and penalty parameters , Construct its augmented Lagrangian function:

[0044] Within the framework of the alternating direction multiplier method, variables are updated iteratively and alternately. and Lagrange multipliers To solve this, the specific iterative steps are as follows: In the first... In the next iteration, first fix , , , Update the restored image .about The subproblems are:

[0045] This quadratic optimization problem achieves efficient computation by solving it in the Fourier domain. Utilizing the Fourier transform properties of convolution and difference, the aforementioned minimization problem is transformed into the Fourier domain, yielding... The frequency domain solution is:

[0046] in, Represents the two-dimensional discrete Fourier transform. Indicates complex conjugation. This represents element-wise multiplication. and These represent the convolution kernel matrices corresponding to the horizontal and vertical difference operators, respectively. , The result can be obtained by calculating the above formula and performing an inverse Fourier transform. Next, fix it. , , , Update auxiliary variables .about The subproblems are:

[0047] This problem has a closed-ended solution and can be solved using the soft threshold operator:

[0048] The soft threshold operator is defined as follows: , For the threshold, For symbolic functions. Similarly, fixed. , , , Update auxiliary variables .about The subproblems are:

[0049] The solution can be obtained using the weighted soft threshold operator:

[0050] The weighted soft thresholding operation is performed independently on each element of the vector, the first... The calculation of each element is as follows: ,in This corresponds to the weight threshold. Finally, update the Lagrange multipliers:

[0051]

[0052] Set the iteration stopping condition to relative error Less than the threshold Or it may reach the maximum number of iterations, 100. When the iterations converge, the last update yields... This is the preliminary restored image.

[0053] The deconvolution process also includes a motion blur point spread function update step. After the first iteration, a preliminary restored image estimate is obtained. Then, fix the image, and within the gradient saliency region, based on The gradient information is used to re-estimate the parameters of the motion blur point spread function. Specifically, the gradient information is calculated. The gradient magnitude map and gradient direction map are obtained, and the structure tensor is recalculated within the gradient saliency region. The initial fuzzy direction is then fine-tuned based on the principal eigenvector direction to obtain the updated fuzzy direction estimate. At the same time, based on Edge information within the gradient saliency region is used to re-estimate the fuzzy length by analyzing the edge profile. Using the updated parameters and Reconstructing the Kernel of Motion Blur Point Spread Function In subsequent iterations of the alternating direction multiplier method, the updated... Used for calculating data fidelity terms, i.e., replacing the original energy function. The process continues iterating until convergence, and finally outputs the optimized image as the initial restored image.

[0054] Step S4: Perform multi-scale frequency domain decomposition on the preliminary restored image to obtain multi-scale frequency domain sub-bands; then perform cross-scale nonlinear fusion and detail reconstruction on the multi-scale frequency domain sub-bands to generate the final restored image.

[0055] In this embodiment, the initially restored image is first subjected to multi-scale pyramid decomposition. The Laplacian pyramid decomposition algorithm is used, and the number of pyramid layers is set to 4. Specifically, the initially restored image is used as the bottom layer of the pyramid. ,right Gaussian low-pass filtering and downsampling are performed to obtain the low-frequency approximation image of the next layer. This process is repeated to generate sequentially. , Together, they form the Pyramid of Gauss. Each level of the Pyramid of Laplace... By calculating the image of this layer of the Gaussian pyramid Compared with its upper layer image Image after upsampling and interpolation The difference is obtained, that is This results in four image layers at different scales. , , , ,in For the finest scale, This is the coarsest scale.

[0056] Next, a multi-resolution transformation is performed on each layer of the pyramid image to decompose it into frequency domain subbands. This embodiment uses shear wave transform as the multi-resolution transformation method. For the first... Layered Laplace Pyramid Image The shear wave transform is applied. This transform first performs multi-scale decomposition using a Laplacian pyramid, and then performs directional analysis using a localized directional filter bank at each scale. In this embodiment, the number of decomposition scales for the shear wave transform is set to 4, and the number of directions is set to 8, 8, 16, and 16 at scales from coarse to fine, respectively. After the transform, each image layer... It is decomposed into a low-frequency subband representing approximate information. And a series of high-frequency subbands representing detailed information in different directions. ,in Indicates scale index. This indicates the direction index. At this point, the multi-scale frequency domain decomposition is complete, yielding multiple sets of frequency domain sub-bands ranging from the finest to the coarsest scale.

[0057] Subsequently, cross-scale nonlinear fusion and detail reconstruction are performed on the multi-scale frequency domain subbands. The core lies in calculating the correlation metric of the transform coefficients at the same spatial location in high-frequency subbands at different scales, and constructing adaptive fusion weights accordingly. For any spatial coordinate... Iterate through all scales except the coarsest one, and calculate its value between two adjacent scales. and Up, in the same direction The correlation measure between the high-frequency subband coefficients. By calculation Centered on, size is The normalized cross-correlation coefficient is obtained from the absolute values ​​of the coefficients within the window, and the calculation formula is as follows:

[0058] in Indicated by A 5x5 local window centered on the pixel is used to sum and iterate through all pixels within the window. Association metrics The value is between 0 and 1, reflecting the degree of consistency of the feature at different scales.

[0059] Based on the calculated correlation metric An adaptive weighting mechanism is constructed to fuse high-frequency subband coefficients at different scales. For the first... The scale coefficients, their fusion weights The coefficient's location is determined by its coarser scale relative to its neighbors. The correlation metric at the corresponding location determines, specifically: ,in yes function, . The function maps association metrics to weight values ​​between 0 and 1, with weights closer to 1 for high association metrics and closer to 0 for low association metrics. It also represents the coarsest scale. The high-frequency subband coefficients are assigned a fixed weight of 0.5.

[0060] Using calculated adaptive weights, high-frequency subband coefficients at at least two different scales are fused. This embodiment employs a layer-by-layer fusion strategy from fine to coarse scales. First, the fused high-frequency subband coefficient set is initialized. For the finest scale Each high frequency coefficient Its fusion value is calculated as follows:

[0061] in for exist The corresponding coordinates on the scale. Next, for the scale... Its coefficients, which may have been used as supplements in the previous round, are updated to their fusion values ​​in this round as follows:

[0062] This process continues at coarser scales until high-frequency coefficients at all scales participate in the fusion, ultimately resulting in a fused and enhanced set of multi-scale, multi-directional high-frequency subbands. .

[0063] Finally, based on the fused high-frequency subband and the low-frequency subband at the top layer of the pyramid (the coarsest scale), multi-resolution inverse transform and pyramid reconstruction are performed to generate the final restored image. First, for each scale... The high-frequency subbands in all directions of this scale will be fused. Compared with the original low-frequency approximate subband (For the coarsest scale) or the upsampled reconstruction result from the previous scale (for other scales), a shear wave inverse transform is performed to reconstruct the spatial domain image at that scale. This process starts at the coarsest scale, upsamples its inverse transform result, and uses it as the low-frequency input for the next finer scale inverse transform, proceeding sequentially until the image at the finest scale is reconstructed. Finally, this reconstructed image at the finest scale is added to the residual from the Laplacian pyramid decomposition process to obtain the final sharp image after motion blur suppression and detail restoration.

[0064] Example 2: The present invention provides a method for motion blur suppression and detail restoration of images for remote observation by UAVs, which also includes preprocessing the original video stream before acquiring the degraded image.

[0065] The preprocessing steps include performing a deep learning-based spatiotemporal joint denoising operation on each frame of the original video stream to suppress thermal noise and quantization noise from the image sensor.

[0066] Specifically, a lightweight spatiotemporal convolutional neural network is used, taking three consecutive frames of images as input and outputting the denoising result of the current frame.

[0067] Subsequently, phase-correlation-based subpixel-level inter-frame registration was performed on the denoised video frame sequence to eliminate high-frequency micro-displacements caused by drone hovering jitter.

[0068] The registered frame sequence is then further enhanced by motion-compensated temporal filtering to improve the signal-to-noise ratio, providing a higher quality input image for subsequent motion blur suppression and detail restoration.

[0069] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for motion blur suppression and detail restoration in remote observation images from unmanned aerial vehicles (UAVs), characterized in that, include: Step S1: Acquire a degraded image containing motion blur from the drone during flight; Step S2: In the degraded image, based on gradient detection, determine the gradient saliency region, and fuse the anisotropic features of the local structure of the image to adaptively estimate the motion blur point spread function. Step S3: Based on the motion blur point spread function, construct a regularized deconvolution model with edge protection constraints, and perform deconvolution processing on the degraded image to obtain a preliminary restored image; wherein, the data fidelity term of the regularized deconvolution model is used to characterize the convolution relationship between the degraded image, the motion blur point spread function and the image to be restored, and its constraint term includes a total variation regularization sub-term and a spatially weighted gradient constraint sub-term; Step S4: Perform multi-scale frequency domain decomposition on the preliminary restored image to obtain multi-scale frequency domain sub-bands; then perform cross-scale nonlinear fusion and detail reconstruction on the multi-scale frequency domain sub-bands to generate the final restored image.

2. The method for motion blur suppression and detail restoration of UAV remote observation images as described in claim 1, characterized in that, Step S1 includes: Receive the raw video stream transmitted by the image sensor of the drone while it is in flight; Based on the real-time motion data provided by the UAV flight control system, estimate the global motion trend between frames; By combining global motion trends, the pixel displacement vector field between adjacent frames in the original video stream is calculated; Based on the statistical characteristics of the pixel displacement vector field, degraded image frames containing significant motion blur are automatically selected from the original video stream.

3. The method for motion blur suppression and detail restoration of UAV remote observation images as described in claim 1, characterized in that, In step S2, determining the gradient significance region specifically involves: Calculate the gradient magnitude map and gradient direction map of the degraded image; On the gradient magnitude map, the variance of the gradient magnitude in the neighborhood of each pixel is calculated using a sliding window method. On the gradient direction map, the angular consistency measure of the gradient direction is calculated using the same sliding window method. Pixel regions with gradient magnitude variance greater than the first threshold and gradient direction consistency higher than the second threshold are identified as gradient saliency regions.

4. The method for motion blur suppression and detail restoration of UAV remote observation images as described in claim 3, characterized in that, The anisotropic features of the local structure of the fused image are specifically as follows: Within the gradient saliency region, calculate the structure tensor matrix of each image patch and solve for its principal eigenvalues, secondary eigenvalues, and principal eigenvector directions; Calculate the anisotropy coefficients based on the principal and secondary eigenvalues; Based on the direction of the principal eigenvector, the initial fuzzy direction estimate obtained from the gradient magnitude map and Radon transform is corrected.

5. The method for motion blur suppression and detail restoration of UAV remote observation images as described in claim 4, characterized in that, The adaptive estimation motion blur point diffusion function is specifically as follows: Based on the corrected fuzzy direction and anisotropic coefficients, a motion fuzzy point diffusion function kernel model is constructed. By analyzing the energy diffusion width of the Radon transform projection curve at the corresponding corrected angle, the final fuzzy length estimate is obtained, thus completing the adaptive estimation of the motion fuzzy point diffusion function.

6. The method for motion blur suppression and detail restoration of UAV remote observation images as described in claim 1, characterized in that, In step S3, the regularized deconvolution model is represented as a minimization problem of the following energy function: in, For degraded images, Let be the motion blur point spread function. For the image to be restored, This represents the convolution operation. Denotes the total variation regularity term. Represents the gradient operator, The spatial variation weight matrix is ​​derived from the image content. This represents the predefined norm function. and This is the regularization parameter.

7. The method for motion blur suppression and detail restoration of UAV remote observation images as described in claim 6, characterized in that, The deconvolution processing of the degraded image specifically includes: By employing a variable splitting strategy and introducing auxiliary variables, the problem of minimizing the energy function is decomposed into subproblems concerning the restored image and each auxiliary variable. Within the framework of the alternating direction multiplier method, each subproblem is solved iteratively. The subproblems concerning the restored image are solved in the Fourier domain, while the subproblems involving regularization constraints are solved using the corresponding proximal operators.

8. The method for motion blur suppression and detail restoration of UAV remote observation images as described in claim 7, characterized in that, The deconvolution process includes a motion blur point spread function update step: After completing one or more iterations, fix the current restored image, and within the gradient saliency region, re-estimate the parameters of the motion blur point spread function based on the gradient information of the current restored image; The updated motion blur point diffusion function is used to calculate the data fidelity term in subsequent iterations.

9. The method for motion blur suppression and detail restoration of UAV remote observation images as described in claim 1, characterized in that, In step S4, the preliminary restored image undergoes multi-scale frequency domain decomposition, specifically as follows: The initially restored image is decomposed into multiple scale pyramids to obtain multiple image layers of different scales. For each layer of the pyramid image, a multi-resolution transformation is performed to decompose it into low-frequency sub-bands and high-frequency sub-bands in multiple directions; Multi-resolution transforms include wavelet transform, contour wave transform, or shear wave transform.

10. The method for motion blur suppression and detail restoration of UAV remote observation images as described in claim 9, characterized in that, The cross-scale nonlinear fusion and detail reconstruction of multi-scale frequency domain subbands specifically includes: Calculate the correlation measure of the transform coefficients of the same spatial location in high-frequency subbands at different scales; Based on the correlation metric, an adaptive weight is constructed for fusing high-frequency subband coefficients at different scales; Using adaptive weights, the high-frequency subband coefficients of at least two different scales are fused; Based on the fused high-frequency subband and the low-frequency subband at the top of the pyramid, multi-resolution inverse transform and pyramid reconstruction are performed to obtain the final restored image.

Images