Monocular vision image defuzzification method for low-altitude miniature unmanned aerial vehicle

Abstract

The invention discloses a monocular vision image deblurring method for a low-altitude miniature unmanned aerial vehicle, which belongs to the field of miniature unmanned aerial vehicle image processing, and comprises the following steps: preprocessing a blurred image to obtain a preprocessed blurred image and a blurred kernel estimation result; constructing a deep reinforcement learning model based on an attention mechanism and memory playback, and dividing the preprocessed blurred image into overlapped sub-image blocks to obtain a plurality of blurred image blocks and corresponding blurred kernel blocks; inputting the blurred image blocks and the corresponding blurred kernel blocks into a deep reinforcement learning model, performing channel-level fusion on the attention weight map, the blurred image blocks and the blurred kernel blocks, performing deblurring processing according to a dual-network structure, and outputting deblurred sub-blocks consistent with the input size; and based on the deblurred sub-blocks, eliminating the discontinuity of the boundaries of the sub-blocks through weighted fusion of overlapped regions to obtain a reconstructed image. According to the method, the problem of edge fault easily occurring in a traditional blocking method is solved, and the processing speed and the image quality in a complex scene are both considered.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of micro unmanned aerial vehicle image processing, and particularly relates to a monocular vision image deblurring method for low-altitude micro unmanned aerial vehicles. BACKGROUND

[0002] Low-altitude micro unmanned aerial vehicles have wide application prospects in aerial photography, environmental monitoring, agricultural plant protection, emergency rescue and other fields. However, due to the complex and changeable low-altitude environment, such as atmospheric turbulence, unmanned aerial vehicle vibration, unstable lighting conditions and fast target movement, the images obtained by the monocular vision system often appear blurred, which seriously affects the quality of the images and the accuracy of subsequent target detection, recognition, tracking and other tasks.

[0003] Traditional image deblurring methods mainly include model-based methods and statistical-based methods. The model-based method usually assumes that the image blur is caused by a linear shift-invariant system, estimates the blur kernel, and then uses inverse filtering, Wiener filtering and other methods to restore the blurred image. However, in the complex low-altitude environment, the blur process is often nonlinear and shift-variant, and the traditional model-based method is difficult to accurately estimate the blur kernel, resulting in poor deblurring effect. The statistical-based method learns a large number of clear-blurred image pairs to establish a mapping relationship from blurred images to clear images. Early statistical methods such as support vector machines and sparse coding have limited feature extraction capabilities, and the deblurring effect is also limited. With the development of deep learning technology, existing deep learning-based deblurring methods still have some problems: a large amount of training data is usually required, but it is difficult to obtain a large number of clear-blurred image pairs in the actual application of low-altitude micro unmanned aerial vehicles; moreover, the performance of existing methods in dealing with complex blur scenes such as dynamic blur and multi-motion direction blur still needs to be improved. In summary, the adaptability and deblurring effect of the existing monocular vision image deblurring method for low-altitude micro unmanned aerial vehicles in complex environments still need to be improved, therefore, there is an urgent need for a monocular vision image deblurring method for low-altitude micro unmanned aerial vehicles that can effectively handle complex blur scenes and reduce the dependence on training data. SUMMARY

[0004] To solve the above technical problems, the application provides a monocular vision image deblurring method for low-altitude micro unmanned aerial vehicles to solve the problems existing in the prior art.

[0005] To achieve the above purpose, the application provides a monocular vision image deblurring method for low-altitude micro unmanned aerial vehicles, comprising: The blurred image obtained by the low-altitude micro unmanned aerial vehicle is preprocessed to obtain a preprocessed blurred image and a blur kernel estimation result; construct a deep reinforcement learning model based on an attention mechanism and memory playback, divide the preprocessed blurred image into overlapping sub-image blocks to obtain a plurality of blurred image blocks, and determine a blur kernel block corresponding to each blurred image block based on the blur kernel estimation result; input the blurred image blocks and the corresponding blur kernel blocks into the deep reinforcement learning model, the deep reinforcement learning model performs channel-level fusion of an attention weight map and local features of the blurred image blocks and the blur kernel blocks, performs deblurring processing according to a double network structure, and outputs deblurred sub-blocks with the same size as the input; based on the deblurred sub-blocks, eliminate the discontinuity of the block boundary by overlapping area weighted fusion to obtain a reconstructed image.

[0006] Optionally, the process of preprocessing the blurred image obtained by the low-altitude miniature unmanned aerial vehicle includes: normalizing and calibrating the blurred image to obtain a normalized image; performing median filtering and noise reduction processing on the normalized image to obtain a preprocessed blurred image; performing blur kernel estimation on the preprocessed blurred image based on an improved variational model to obtain a blur kernel estimation result.

[0007] Optionally, the improved variational model is: ; wherein, denotes the L2 norm of and denote the L1 norm of the clear image gradient and the blur kernel gradient, respectively, is a first regularization parameter, is a second regularization parameter, is a clear image, is a blur kernel block, denotes convolution operation, denotes a gradient operator, is a preprocessed blurred image.

[0008] Optionally, the process in which the deep reinforcement learning model performs channel-level fusion of an attention weight map and local features of blurred image blocks and blur kernel blocks and performs deblurring processing according to action selection of a policy network includes: input the blurred image blocks and the corresponding blur kernel blocks into the deep reinforcement learning model, the deep reinforcement learning model extracts features of the blurred image blocks and the blur kernel blocks through a convolution layer; obtain normalized features of the attention weight map based on the attention weight map; The features of the blurred image patch and the blurred kernel patch are concatenated with the normalized features of the attention weight map to obtain the state; The dual-network structure performs action selection and value evaluation on the state to obtain deblurred sub-blocks.

[0009] Optionally, the dual-network structure includes: a policy network and a value network; The process of obtaining deblurred sub-blocks by performing action selection and value evaluation on the state based on the dual-network structure includes: The policy network outputs the action probability distribution based on the state; The value network evaluates the long-term expected reward of the current state based on the state, and obtains the value evaluation result. Based on the action probability distribution and value assessment results, the blurred image block is deblurred, and the deblurred sub-block with the same size as the input is output.

[0010] Optionally, the expression for calculating the probability distribution of actions is: ; In the formula, exp is an exponential function. In response to The convolutional feature extraction function, In response to The convolutional feature extraction function, For the action space, In the action space, except All other possible actions besides For state, According to The chosen action Let be the probability of the action.

[0011] Optionally, the deep reinforcement learning model is trained using a memory replay training mechanism; the process of training the model using the memory replay training mechanism includes updating the training loss function of the value network and optimizing the policy network. The expression for calculating the training loss function of the value network is as follows: ; In the formula, Let be the training loss function for the value network. Represents the experience replay buffer Medium sampling The expected result The result of the state valuation. This is a discount reward.

[0012] Optionally, the objective of optimizing the policy network is to maximize the expected reward; The objective function of the policy network is: ; In the formula, The objective function of the policy network is represented. For the parameters of the policy network, Representing state Obedience by strategy Induced state distribution , Indicates action Obedience Policy Network The probability distribution of action selection. for Expectations This is the reward function.

[0013] Optionally, several blurred image blocks and corresponding blurred kernel blocks are input into the deep reinforcement learning model to obtain the expression for the deblurred sub-blocks: ; In the formula, This represents the row index of the image patch in a two-dimensional grid. Indicates column index, For deep reinforcement learning networks based on attention and memory playback, For the action A defined set of network parameters To extract from the preprocessed blurred image The extracted first Overlapping sub-images, The input is used to obtain the deblurred sub-image from the ADRL network.

[0014] Optionally, based on the deblurred sub-blocks, the discontinuities at the block boundaries are eliminated through weighted fusion of overlapping regions, resulting in the following expression for the reconstructed image: ; In the formula, Represents the coordinates in the reconstructed image Pixel value at that location, This indicates that for all pixels... Image Patch Index Weighted fusion weight Summation, Relative coordinates within the block, and For the first The coordinates of the top left corner of each image patch.

[0015] Compared with the prior art, the present invention has the following advantages and technical effects: The monocular vision image deblurring method of the application is aimed at blurred images obtained by low-altitude micro unmanned aerial vehicles in complex environments, and proposes an efficient and adaptive deblurring solution. Through the preprocessing step, the blurred image is normalized, calibrated, denoised and blurred kernel estimated, providing high-quality input for subsequent deblurring processing. Combined with a deep reinforcement learning model based on attention mechanism and memory replay, it can dynamically focus on the key areas in the image and intelligently adjust the deblurring strategy according to the local blur characteristics. In addition, the overlapping block and weighted fusion technology is adopted, effectively solving the discontinuity problem of the block boundary and improving the overall quality of the reconstructed image. This method not only significantly improves the image deblurring effect, but also enhances the adaptability and generalization ability of the model in complex scenes, providing clearer and more reliable visual data support for the application of low-altitude micro unmanned aerial vehicles in marine ecological environment monitoring and other fields. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which form a part of this application, are intended to provide further understanding of the application and are incorporated herein in their entirety, and the illustrative embodiments of the application and their description serve to explain the application. The drawings in the accompanying drawings are as follows: Figure 1 The flow chart of the monocular vision image deblurring method for low-altitude micro unmanned aerial vehicles of the embodiments of the application. DETAILED DESCRIPTION

[0017] It should be noted that the embodiments and features in the embodiments of the application can be combined with each other without conflict. The application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0018] It should be noted that the steps shown in the flow chart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flow chart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0019] Embodiment one The application relates to a monocular vision image deblurring method suitable for low-altitude micro unmanned aerial vehicles, which comprises the following steps: for the image blurring problem caused by complex environments such as atmospheric turbulence and unmanned aerial vehicle vibration, the original blurred image is subjected to normalization and median filtering pretreatment to eliminate noise interference and pixel value distribution deviation; an improved variational model is adopted to jointly constrain the gradient sparsity of a clear image and a blur kernel, so that the blur kernel estimation accuracy is improved; a deep reinforcement learning model (ADRL) based on an attention mechanism and memory playback is constructed, the attention weight map, the image block and the blur kernel block features are fused into a composite state, the double-network structure of the shared feature extraction layer of a policy network and a value network is used to realize adaptive optimization of the deblurring strategy, and the memory playback mechanism is used to improve the training efficiency; finally, the large-size image is subjected to overlapping block processing, the ADRL network is used to independently process each sub-block, the boundary discontinuity problem is eliminated through weighted fusion, and the globally consistent clear image reconstruction is realized. Through the combination of attention focusing and dynamic strategy adjustment, the accurate estimation of the blur kernel and the efficient recovery of image details under complex environments are realized.

[0020] As shown in Figure 1 , in the embodiment, a monocular vision image deblurring method for low-altitude micro unmanned aerial vehicles is provided, which combines a deep reinforcement learning model (ADRL) based on an attention mechanism and memory playback, can effectively process the monocular vision image blurring problem under low-altitude complex environments, and improves the image deblurring effect and the adaptability of the model.

[0021] The image preprocessing enhances the image quality, and comprises steps 100-102.

[0022] In step 100, normalization calibration is performed: in order to avoid the influence of pixel value range difference on neural network training, the blurred image obtained by the low-altitude micro unmanned aerial vehicle is normalized, the image pixel value is normalized to the range of [0, 1], the input of each layer of the neural network is ensured to have a similar numerical range, the convergence speed is accelerated, and the training stability is improved. The normalization formula is: ; Wherein, is the pixel value at the coordinate in the image, is the normalized , is the maximum pixel value in the image, is the minimum pixel value.

[0023] In step 101, the median filtering denoising is performed: a nonlinear filtering method is adopted, the median value is taken after the pixel values in the local window of the image are sorted, the impulse interference such as salt and pepper noise is effectively suppressed, the image background is purified while the edge details are protected, and the input quality of the blur kernel estimation is improved.

[0024] For the common noise in low-altitude environment, median filtering is used for noise reduction to obtain the preprocessed blurred image . For a rectangular filter window with size , the median filtering method is used for noise filtering, ; wherein, is the pixel value at coordinate in the preprocessed image, is the set obtained by arranging the pixel values in ascending order.

[0025] Step 102, variational model blur kernel estimation: based on the blur image formation model, the blur process is modeled as the convolution of the clear image and the blur kernel plus noise, the data consistency is constrained by minimizing the L2 norm error, the overfitting is suppressed by combining the L1 norm gradient regularization, and the local smoothness of the clear image and the blur kernel is ensured.

[0026] An improved variational model is used to estimate the blur kernel of the preprocessed blurred image, and the variational model is constructed as follows: ; wherein, denotes the L2 norm of , and denote the L1 norm of the clear image gradient and the blur kernel gradient, respectively. is the first regularization parameter, is the second regularization parameter, is the clear image, is the blur kernel block, denotes the convolution operation. denotes the gradient operator.

[0027] The process of constructing the intelligent deblurring network based on ADRL includes steps 200-204.

[0028] Step 200, construct an attention and memory replay deep reinforcement learning (ADRL) model, in the state space construction, the attention weight map is fused with the local features of the image block and the blur kernel block at the channel level to form a composite state representation containing spatial structure, blur characteristics and regional importance, providing multi-dimensional information input for policy decision.

[0029] Specifically, for the preprocessed image block and the blur kernel block m, the features and , and then spliced with the normalized features of the attention weight map , to obtain a state : ; wherein, is an image splicing operation.

[0030] Step 201, double network structure design: policy network and value network , the architecture of the policy network and the value network shares the feature extraction layer. The policy network outputs the action probability distribution based on the composite state, and the value network estimates the long-term expected reward of the current state. The design of the shared feature extraction layer not only reduces the parameter redundancy, but also strengthens the correlation modeling of local features and deblurring strategies, improving the model's representation ability for complex scenes.

[0031] The policy network is used to select an action according to the state , and the value network is used to estimate the value of the state . Both the policy network and the value network use a multi-layer convolutional neural network structure, wherein the policy network uses a Softmax activation function in the output layer to output the probability of each action; ; wherein, exp is the exponential function, is a convolutional feature extraction function for , is a convolutional feature extraction function for , is an action space, is all other possible actions in the action space except .

[0032] Step 202, the value network uses a linear activation function in the output layer to output the value of the state.

[0033] ; wherein, is a mapping function. The policy network and the value network share the convolutional feature extraction layer to improve the parameter utilization efficiency, and the subsequent branches process the policy output and the value estimation respectively. This structure design can effectively capture the correlation between local features of image blocks and deblurring strategies.

[0034] Step 203, memory replay training mechanism: store historical experience tuples through the memory replay unit, and randomly sample batch data for parameter update during training. This mechanism breaks the strong correlation between consecutive data, avoids the model from losing generalization ability due to overfitting to the current scene, and significantly improves the training stability and strategy optimization efficiency.

[0035] Memory playback unit Stored experience tuples Batch sampling is used during training, where... For the reward function, for The state at any given time. Training is performed using randomly sampled historical experience data to reduce correlations between data points and improve the model's training efficiency and stability. The training loss function for the value network. for: ; in, As a discount reward, Represents the experience replay buffer Medium sampling The expected outcome.

[0036] Step 204: The policy network is optimized using the policy gradient method, with the goal of maximizing the expected reward. ; in, The objective function of the policy network is represented. For the parameters of the policy network, Representing state Obedience by strategy Induced state distribution . Indicates action Obedience Policy Network The probability distribution of action selection. for The expectation.

[0037] The image deblurring process includes steps 300-301.

[0038] Step 300, Block Processing: The preprocessed blurred image is divided into overlapping sub-image blocks. Each sub-image block is independently input into the ADRL model for deblurring, and the output is a deblurred sub-block with the same size as the input. Block processing reduces computational complexity and allows the model to dynamically adjust its strategy for sub-blocks with different degrees of blur.

[0039] Considering the large image size and uneven blur level of low-altitude micro UAV images, the images are divided into overlapping image blocks, and the pre-processed blurred images are analyzed. The extracted first Extracting sub-images from overlapping sub-images The input to the ADRL network yields the deblurred sub-image. The sub-images output by the network have the same size as the input, maintaining the spatial correspondence.

[0040] ; in, This represents the row index of the image patch in a two-dimensional grid. Indicates column index, For deep reinforcement learning networks based on attention and memory playback, For the action A defined set of network parameters.

[0041] Step 301, Overlap Fusion: Discontinuities at block boundaries are eliminated through weighted fusion of overlapping regions. During the fusion process, the final value of each pixel is obtained by a weighted sum of the deblurred results of all sub-blocks containing that pixel. The weights are dynamically adjusted based on the pixel's position within the sub-block (e.g., lower weights for edge regions and higher weights for center regions). This strategy ensures the global continuity and detail consistency of the reconstructed image.

[0042] After deblurring each image patch independently, the overlapping areas are merged to eliminate the discontinuity of the patch boundaries, thus obtaining a complete and clear reconstructed image. The specific formula is as follows: ; in, Represents the coordinates in the reconstructed image Pixel value at that location, For all pixels Image Patch Index Weighted fusion weight Perform summation. For the first Deblurring results for individual image patches Relative coordinates within the block, and For the first The coordinates of the top left corner of each image patch.

[0043] Through the above steps, this invention achieves efficient deblurring of monocular vision images of low-altitude micro UAVs. It can adaptively adjust the deblurring strategy in complex environments such as atmospheric turbulence and UAV vibration, effectively improving the accuracy of blur kernel estimation and the ability to restore image details.

[0044] This invention overcomes the limitations of traditional single-variable constraints by jointly constraining the gradient sparsity of the sharp image and the blur kernel through an improved variational model, accurately capturing the coupling relationship between the blur kernel and image content. This design effectively suppresses artifacts and constrains the structural complexity of the blur kernel, enabling the model to adaptively optimize blur kernel estimation in complex scenarios such as atmospheric turbulence and UAV vibration. This significantly improves the ability to recover details such as image edges and textures, providing a clearer and more realistic image foundation for subsequent visual tasks.

[0045] The present application is directed to the large size image characteristics of the unmanned aerial vehicle, adopts the overlapping block input and the weighted fusion strategy, reduces the calculation complexity, and eliminates the discontinuity problem of the block boundary. Combined with the dynamic focusing of the attention mechanism to the high blur area and the adaptive adjustment of the ADRL strategy, the model can intelligently optimize the deblurring operation for local areas with different blur degrees, which not only ensures the processing efficiency, but also realizes the global smooth transition of the reconstructed image, effectively solves the edge fault problem easily appeared in the traditional block method, and balances the processing speed and image quality in the complex scene.

[0046] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for deblurring monocular vision images for low-altitude micro unmanned aerial vehicles, characterized in that, Includes the following steps: The blurred images acquired by the low-altitude micro UAV are preprocessed to obtain the preprocessed blurred images and the blur kernel estimation results; A deep reinforcement learning model based on attention mechanism and memory replay is constructed. The preprocessed blurred image is divided into overlapping sub-image blocks to obtain several blurred image blocks. The blurred kernel block corresponding to each blurred image block is determined based on the blurred kernel estimation result. The blurred image patch and the corresponding blurred kernel block are input into the deep reinforcement learning model. The deep reinforcement learning model performs channel-level fusion of the attention weight map with the local features of the blurred image patch and the blurred kernel block, and then performs deblurring processing according to the dual network structure, outputting a deblurred sub-block with the same size as the input. Based on the deblurred sub-blocks, the discontinuities at the block boundaries are eliminated by weighted fusion of overlapping regions to obtain the reconstructed image.

2. The monocular vision image deblurring method for low-altitude micro UAVs according to claim 1, characterized in that, The preprocessing of blurry images acquired by low-altitude micro UAVs includes: The blurred image is normalized to obtain a normalized image; The normalized image is subjected to median filtering for noise reduction to obtain a preprocessed blurred image; The preprocessed blurred image is subjected to fuzzy kernel estimation based on the improved variational model to obtain the fuzzy kernel estimation result.

3. The monocular vision image deblurring method for low-altitude micro UAVs according to claim 2, characterized in that, The improved variational model is as follows: ； In the formula, express L2 norm, and Let L1 norms represent the gradients of the sharp image and the blur kernel, respectively. The first regularization parameter is used. This is the second regularization parameter. For a clear image, For fuzzy kernel blocks, This represents the convolution operation. Represents the gradient operator. This is the blurred image after preprocessing.

4. The monocular vision image deblurring method for low-altitude micro UAVs according to claim 1, characterized in that, The deep reinforcement learning model performs channel-level fusion of the attention weight map with the local features of blurred image patches and blurred kernel patches. The deblurring process based on the action selection of the policy network includes: The blurred image patch and the corresponding blurred kernel block are input into the deep reinforcement learning model, and the deep reinforcement learning model extracts the features of the blurred image patch and the blurred kernel block through convolutional layers; The normalized features of the attention weight map are obtained based on the attention weight map. The features of the blurred image patch and the blurred kernel patch are concatenated with the normalized features of the attention weight map to obtain the state; The dual-network structure performs action selection and value evaluation on the state to obtain deblurred sub-blocks.

5. The monocular vision image deblurring method for low-altitude micro UAVs according to claim 4, characterized in that, The dual-network structure includes: a policy network and a value network; The process of obtaining deblurred sub-blocks by performing action selection and value evaluation on the state based on the dual-network structure includes: The policy network outputs the action probability distribution based on the state; The value network evaluates the long-term expected reward of the current state based on the state, and obtains the value evaluation result. Based on the action probability distribution and value assessment results, the blurred image block is deblurred, and the deblurred sub-block with the same size as the input is output.

6. The monocular vision image deblurring method for low-altitude micro UAVs according to claim 5, characterized in that, The expression for calculating the probability distribution of actions is: ； In the formula, exp is an exponential function. In response to The convolutional feature extraction function, In response to The convolutional feature extraction function, For the action space, In the action space, except All other possible actions besides For state, According to The chosen action Let be the probability of the action.

7. The monocular vision image deblurring method for low-altitude micro UAVs according to claim 6, characterized in that, The deep reinforcement learning model is trained using a memory replay training mechanism; the process of training the model using the memory replay training mechanism includes updating the training loss function of the value network and optimizing the policy network. The expression for calculating the training loss function of the value network is as follows: ； In the formula, Let be the training loss function for the value network. Represents the experience replay buffer Medium sampling The expected result The result of the state valuation. This is a discount reward.

8. The monocular vision image deblurring method for low-altitude micro UAVs according to claim 7, characterized in that, The goal of optimizing the policy network is to maximize the expected reward; The objective function of the policy network is: ； In the formula, This represents the objective function of the policy network. For the parameters of the policy network, Representing state Obedience by strategy Induced state distribution , Indicates action Obedience to policy network The probability distribution of action selection. for Expectations This is the reward function.

9. The monocular vision image deblurring method for low-altitude micro UAVs according to claim 1, characterized in that, By inputting several blurred image blocks and their corresponding blurred kernel blocks into the deep reinforcement learning model, the expression for the deblurred sub-block is obtained as follows: ； In the formula, This represents the row index of the image patch in a two-dimensional grid. Indicates column index, For deep reinforcement learning networks based on attention and memory playback, For the action A defined set of network parameters To extract from the preprocessed blurred image The extracted first Overlapping sub-images, The input is used to obtain the deblurred sub-image from the ADRL network.

10. The monocular vision image deblurring method for low-altitude micro unmanned aerial vehicles according to claim 9, characterized in that, Based on the deblurred sub-blocks, the discontinuities at the block boundaries are eliminated through weighted fusion of overlapping regions, resulting in the following expression for the reconstructed image: ； In the formula, Represents the coordinates in the reconstructed image Pixel value at that location, This indicates that for all pixels... Image Patch Index Weighted fusion weight Summation, Relative coordinates within the block, and For the first The coordinates of the top left corner of each image patch.

Images