Unmanned aerial vehicle image stability enhancement method based on physical guidance

Abstract

The invention discloses an unmanned aerial vehicle image stability enhancement method based on physical guidance, which relates to the technical field of image enhancement and comprises IMU sensor data fusion, physical guidance motion estimation, adaptive image motion compensation and lightweight machine learning optimization. According to the method, firstly, attitude drift errors are corrected by using IMU data, and then global image stabilization transformation is calculated by combining a visual transformation matrix, so that more accurate jitter compensation is realized; and finally, through adaptive image compensation and lightweight machine learning optimization, frame skipping feeling and boundary loss are reduced, and the image stabilization effect and the calculation efficiency are improved. According to the method, IMU sensor data, physical motion modeling and adaptive image compensation are combined, so that the influence of flight jitter of the unmanned aerial vehicle on the image is effectively reduced, and the image stabilization precision and the calculation efficiency are improved; compared with a traditional method, the method has the advantages that the physical consistency of image stabilization is enhanced, the frame skipping feeling and the edge cutting loss are reduced, and the method is suitable for aerial photography, surveying and mapping and reconnaissance tasks of the high-dynamic unmanned aerial vehicle in a complex environment.

Description

Technical Field

[0001] The present invention relates to the technical field of image enhancement, and in particular to a method for enhancing the image stability of an unmanned aerial vehicle based on physical guidance. Background Art

[0002] When UAVs fly in complex environments, they are affected by airflow disturbances, attitude changes, and control errors, which cause image jitter and affect target recognition and mapping accuracy.

[0003] The current methods for enhancing the image stability of drones are mainly divided into two categories: mechanical stabilization and electronic stabilization. Mechanical stabilization relies on the gimbal structure and compensates for the attitude changes of the drone through gyroscopes and motor drives. It is suitable for low-speed stable flight scenarios, but has limited response speed in high-dynamic flight. Electronic stabilization estimates the image motion trajectory through visual algorithms and uses filtering and transformation to compensate for jitter. It has strong flexibility but is susceptible to cumulative errors. Traditional stabilization methods include optical flow, feature matching, Kalman filtering, and L1 trajectory optimization, but they still have the problem of insufficient robustness in complex environments. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a UAV image stability enhancement method based on physical guidance, which can effectively reduce the impact of UAV flight jitter on the image and improve image stabilization accuracy and calculation efficiency.

[0005] The technical solution adopted to solve the above technical problems is: a method for enhancing the image stability of a UAV based on physical guidance, comprising the following steps: S1, IMU sensor data fusion: obtaining the attitude and displacement information of the UAV through IMU data fusion; S2, physical guided motion estimation: performing spatial transformation of the image through a rotation matrix; S3, adaptive image motion compensation: optimizing video stability through adaptive image motion compensation; S4, lightweight machine learning optimization: avoiding edge cropping and enhancing edge details through lightweight machine learning optimization.

[0006] Furthermore, the method for obtaining the attitude and displacement information of the drone through IMU data fusion in S1 is: S101, collect IMU data: accelerometer measures linear acceleration , , , is the linear acceleration along the x-axis, is the linear acceleration along the y-axis, is the linear acceleration along the z-axis, and the gyroscope measures the angular velocity , , , is the angular velocity around the x-axis, is the angular velocity around the y-axis, is the angular velocity around the z axis; S102, calculate the rotation angle: , In the above formula, is the rotation angle at the current moment, is the rotation angle at the previous moment, is the angular velocity at the current moment and , is the time interval, is the drift error at time t, which is corrected by adaptive filtering; , In the above formula, is the adaptive weight, is the rotation angle calculated by the gyroscope, is the rotation angle calculated by the accelerometer, is the drift error at time t-1; , In the above formula, is the standard deviation of the gyroscope data, is the standard deviation of the accelerometer data; S103, calculate the displacement of the drone: , In the above formula, is the position of the UAV at the current time t, is the position of the drone at the last moment t-1, is the speed of the drone at the last moment t-1, is the acceleration of the drone at the current time t and , is the time interval, For compensation items; , In the above formula, is the weight coefficient, is the past time index, is the attenuation factor, is the acceleration of the kth frame, is the average acceleration of the past N frames, is the base of natural logarithms.

[0007] Furthermore, the method of physically guided motion estimation in step S2 is: S201, calculate the rotation transformation matrix of the drone camera, which is the rotation state of the camera at time t: , In the above formula, is the rotation transformation matrix of the drone camera at time t, is the matrix for rotation around the z-axis, is the rotation angle around the Z axis, , is the matrix for rotation around the y axis, is the rotation angle around the Y axis, , is the matrix for rotation around the x-axis, is the rotation angle around the X axis, ; , S202, calculate the final motion transformation matrix: , In the above formula, is the final transformation matrix at time t, is the visual transformation matrix.

[0008] Furthermore, the method of adaptive image motion compensation in step S3 is: S301, calculate compensation error: , In the above formula, is the predicted compensation error, which is used to adjust the transformation matrix of the next frame. , is the weight coefficient, Used to balance the first-order effects, Used to balance the impact of second-order differences, , It is obtained through ridge regression training, and its value is in the range of [-10,10]. is the first-order difference, is the difference in transformation matrix between the current frame and the previous frame, is the second-order difference, and is the change in the first-order difference; , In the above formula, is the final transformation matrix of the current frame, , is the final transformation matrix of the previous frame; S302, calculate the optimized transformation matrix: , In the above formula, is the final transformation matrix at time t, is the predicted compensation error, It is an optimized transformation matrix used to reduce frame skipping and improve stability.

[0009] Furthermore, the method for lightweight machine learning optimization in S4 is: S401, is the optimized 2x3 affine transformation matrix, which is as follows: , Specifically: , , In the above formula, are linear transformation parameters, which are used to control rotation, reduction, enlargement, and shear transformations. , is the translation parameter, Used to control the translation of the image in the x direction. Used to control the translation amount of the image in the y direction; S402, Boundary Correction: Let the original image be , then the image stabilization process based on the affine transformation function T is expressed as: , The corrected result is: , In the above formula, is the filled image, is the stabilized image, is a boundary mask used to identify the area that needs to be filled. It is a Laplace smoothing operator used to detect edges; Based on adaptive edge detection and local contrast enhancement, it is defined as follows: , In the above formula, is the modulus of the image gradient, which can be calculated by the Sobel operator. is the mean of the local gradient, is the standard deviation of the gradient, is the Sigmoid function, which is used to Normalized to between 0 and 1; S403, calculating the final optimized image: , In the above formula, For the final optimized image, is the predicted compensation error, The filled image.

[0010] The beneficial effects of the present invention are as follows: (1) The present invention first uses IMU data to correct the attitude drift error, and then combines the visual transformation matrix to calculate the global image stabilization transformation to achieve more accurate jitter compensation. Finally, through adaptive image compensation and lightweight machine learning optimization, the frame skipping and boundary loss are reduced, and the image stabilization effect and calculation efficiency are improved.

[0011] (2) The present invention combines IMU sensor data, physical motion modeling, and adaptive image compensation to effectively reduce the impact of drone flight jitter on images and improve image stabilization accuracy and computational efficiency. Compared with traditional methods, this method enhances the physical consistency of image stabilization, reduces frame skipping and edge clipping losses, and is suitable for high-dynamic drone aerial photography, mapping, and reconnaissance missions in complex environments.

[0012] (3) The present invention estimates the attitude and displacement of the UAV based on IMU data, and optimizes the image stabilization calculation by combining physical motion modeling. Adaptive filtering is used to correct the gyroscope drift error, and a displacement compensation term is introduced to correct the acceleration accumulation error, so as to improve the accuracy of short-term motion estimation.

[0013] (4) The present invention combines the rotation matrix with the visual transformation matrix to calculate the global motion transformation matrix to optimize the camera shake compensation of the drone. Through the error compensation strategy trained by ridge regression, the first-order difference and second-order difference are calculated to predict the stabilization error, and the transformation matrix is ​​optimized to reduce the frame skipping effect.

[0014] (5) The present invention identifies the cropped area of ​​the stabilized image based on the adaptive boundary mask of the Sobel operator, and performs boundary repair through Laplace smoothing to reduce information loss. Finally, the stabilized image, error compensation and filling area are combined to calculate the optimized image to ensure the integrity of the image after stabilization and the retention of edge details. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a graph of the PSNR performance of the present invention in the original image, motion supplementation, deblurring and fully optimized algorithm steps.

[0016] Figure 2 is a graph of SSIM performance for the original image, motion supplementation, deblurring, and fully optimized algorithm steps.

[0017] Figure 3 Here is a graph of FPS for the original image, motion complement, deblurring, and fully optimized algorithm steps. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0019] A method for enhancing the image stability of a drone based on physical guidance in this embodiment includes the following steps: S1, IMU sensor data fusion: The attitude and displacement information of the drone is obtained through IMU data fusion, which provides the necessary input for physical guided motion estimation.

[0020] The attitude and displacement information of the drone is obtained through IMU data fusion to reduce the impact of sensor errors on image stabilization. The rotation angle is calculated by gyroscope integration, and the drift error is corrected by adaptive filtering. The accelerometer is used for secondary integration to calculate the displacement, and an innovative displacement compensation term is introduced. To optimize short-term jitter errors. Combining gyroscope and accelerometer data can improve the accuracy of drone camera motion estimation and provide reliable physical prior information for subsequent image stabilization compensation.

[0021] IMU (Inertial Measurement Unit) physically guided drone image stability enhancement provides accurate physical data to adjust the camera attitude by sensing the drone's attitude and motion status in real time, effectively reducing jitter and vibration, and can quickly respond to dynamic changes in flight to ensure the stability of images and videos. IMU-guided stability enhancement does not rely on post-processing, directly improves real-time shooting quality, and is suitable for complex flight environments.

[0022] The method of obtaining the attitude and displacement information of the drone through IMU data fusion is: S101, collect IMU data: accelerometer measures linear acceleration , , , is the linear acceleration along the x-axis, is the linear acceleration along the y-axis, is the linear acceleration along the z-axis, and the gyroscope measures the angular velocity , , , is the angular velocity around the x-axis, is the angular velocity around the y-axis, is the angular velocity around the z axis; S102, calculate the rotation angle: , In the above formula, is the rotation angle at the current moment, is the rotation angle at the previous moment, is the angular velocity at the current moment and , is the time interval, is the drift error at time t, which is corrected by adaptive filtering; , In the above formula, is the adaptive weight, is the rotation angle calculated by the gyroscope, is the rotation angle calculated by the accelerometer, is the drift error at time t-1; , In the above formula, is the standard deviation of the gyroscope data, is the standard deviation of the accelerometer data; S103, calculate the displacement of the drone: , In the above formula, is the position of the UAV at the current time t, is the position of the drone at the last moment t-1, is the speed of the drone at the last moment t-1, is the acceleration of the drone at the current time t and , is the time interval, For compensation items; , In the above formula, is the weight coefficient, is the past time index, is the attenuation factor, is the acceleration of the kth frame, is the average acceleration of the past N frames, is the base of natural logarithms.

[0023] S2, physically guided motion estimation: The spatial transformation of the image is performed through the rotation matrix, and the estimation result is used for adaptive image motion compensation for error correction.

[0024] The image is spatially transformed through the rotation matrix to establish an accurate motion model. The rotation matrices around the x-axis, y-axis, and z-axis are calculated, and the total rotation transformation matrix is ​​obtained by multiplying the above matrices to transform the image from the camera coordinate system to the world coordinate system. By multiplying this rotation transformation matrix with the visual transformation matrix, the final image transformation matrix is ​​calculated for the final correction of the image.

[0025] The method of physics-guided motion estimation is: S201, calculate the rotation transformation matrix of the drone camera, which is the rotation state of the camera at time t: , In the above formula, is the rotation transformation matrix of the drone camera at time t, is the matrix for rotation around the z-axis, is the rotation angle around the Z axis, , is the matrix for rotation around the y axis, is the rotation angle around the Y axis, , is the matrix for rotation around the x-axis, is the rotation angle around the X axis, ; , S202, calculate the final motion transformation matrix: , In the above formula, is the final transformation matrix at time t, is the visual transformation matrix.

[0026] S3, Adaptive Image Motion Compensation: Optimize video stability through adaptive image motion compensation. In order to further optimize stability, lightweight machine learning optimization is introduced for fine-tuning.

[0027] Optimize video stability through adaptive image motion compensation. Calculate the compensation error, combining the transformation matrix difference and second-order difference of the previous frame. These parameters are trained through ridge regression to minimize the error and prevent overfitting. The calculated error is used to optimize the transformation matrix of the current frame, thereby reducing frame skipping and improving video stability, effectively improving the smoothness of image processing and visual experience.

[0028] The method of adaptive image motion compensation is: S301, calculate compensation error: , In the above formula, is the predicted compensation error, which is used to adjust the transformation matrix of the next frame. , is the weight coefficient, Used to balance the first-order effects, Used to balance the impact of second-order differences, , It is obtained through ridge regression training, and its value is in the range of [-10,10]. is the first-order difference, is the difference in transformation matrix between the current frame and the previous frame, is the second-order difference, and is the change in the first-order difference; , In the above formula, is the final transformation matrix of the current frame, , is the final transformation matrix of the previous frame; S302, calculate the optimized transformation matrix: , In the above formula, is the final transformation matrix at time t, is the predicted compensation error, It is an optimized transformation matrix used to reduce frame skipping and improve stability.

[0029] S4, lightweight machine learning optimization: Avoid edge clipping and enhance edge details through lightweight machine learning optimization, correct edge information through boundary filling and Laplace smoothing, avoid clipping, and improve visual quality.

[0030] Avoid edge clipping and enhance edge details through lightweight machine learning optimization. Perform boundary correction, use boundary mask and Laplace smoothing operation to fill the edge area of ​​the stabilized image. Calculate the final optimized image, combining the stabilized image, the predicted compensation error and the filled image to ensure the preservation of edge details and the integrity of the image.

[0031] The methods for lightweight machine learning optimization are: S401, is the optimized 2x3 affine transformation matrix, which is as follows: , Specifically: , , In the above formula, are linear transformation parameters, which are used to control rotation, reduction, enlargement, and shear transformations. , is the translation parameter, Used to control the translation of the image in the x direction. Used to control the translation amount of the image in the y direction; S402, Boundary Correction: Let the original image be , then the image stabilization process based on the affine transformation function T is expressed as: , The corrected result is: , In the above formula, is the filled image, is the stabilized image, is a boundary mask used to identify the area that needs to be filled. It is a Laplace smoothing operator used to detect edges; Based on adaptive edge detection and local contrast enhancement, it is defined as follows: , In the above formula, is the modulus of the image gradient, which can be calculated by the Sobel operator. is the mean of the local gradient, is the standard deviation of the gradient, is the Sigmoid function, which is used to Normalized to between 0 and 1; S403, calculating the final optimized image: , In the above formula, For the final optimized image, is the predicted compensation error, The filled image.

[0032] The quantitative analysis of the image stabilization effect in this embodiment uses PSNR (peak signal-to-noise ratio), SSIM (structural similarity index) and FPS (frame rate). The experimental results are as follows: Figures 1 to 3 As shown in the figure, the algorithm optimization significantly improves the image quality. The increase in PSNR and SSIM shows that the image quality is gradually improving. During the stability enhancement process, the frame rate changes little, ensuring the real-time performance of the algorithm.

[0033] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. A method for enhancing the image stability of a drone based on physical guidance, characterized in that: The following steps are involved: S1, IMU sensor data fusion: obtain the attitude and displacement information of the drone through IMU data fusion; S2, physics-guided motion estimation: spatial transformation of the image via a rotation matrix; S3, Adaptive Image Motion Compensation: Optimize video stability through adaptive image motion compensation; S4, Lightweight Machine Learning Optimization: Avoid edge clipping and enhance edge details through lightweight machine learning optimization.

2. The method for enhancing the image stability of a UAV based on physical guidance according to claim 1, characterized in that: The method for obtaining the attitude and displacement information of the drone through IMU data fusion in S1 is: S101, collect IMU data: accelerometer measures linear acceleration , , , is the linear acceleration along the x-axis, is the linear acceleration along the y-axis, is the linear acceleration along the z-axis, and the gyroscope measures the angular velocity , , , is the angular velocity around the x-axis, is the angular velocity around the y-axis, is the angular velocity around the z axis; S102, calculate the rotation angle: , In the above formula, is the rotation angle at the current moment, is the rotation angle at the previous moment, is the angular velocity at the current moment and , is the time interval, is the drift error at time t, which is corrected by adaptive filtering; ， In the above formula, is the adaptive weight, is the rotation angle calculated by the gyroscope, is the rotation angle calculated by the accelerometer, is the drift error at time t-1; ， In the above formula, is the standard deviation of the gyroscope data, is the standard deviation of the accelerometer data; S103, calculate the displacement of the drone: ， In the above formula, is the position of the UAV at the current time t, is the position of the drone at the last moment t-1, is the speed of the drone at the last moment t-1, is the acceleration of the drone at the current time t and , is the time interval, For compensation items; ， In the above formula, is the weight coefficient, is the past time index, is the attenuation factor, is the acceleration of the kth frame, is the average acceleration of the past N frames, is the base of natural logarithms.

3. The method for enhancing the image stability of a UAV based on physical guidance according to claim 2, characterized in that: The method of physically guided motion estimation in step S2 is: S201, calculate the rotation transformation matrix of the drone camera, which is the rotation state of the camera at time t: ， In the above formula, is the rotation transformation matrix of the drone camera at time t, is the matrix for rotation around the z-axis, is the rotation angle around the Z axis, , is the matrix for rotation around the y axis, is the rotation angle around the Y axis, , is the matrix for rotation around the x-axis, is the rotation angle around the X axis, ; ， S202, calculate the final motion transformation matrix: ， In the above formula, is the final transformation matrix at time t, is the visual transformation matrix.

4. The method for enhancing the image stability of a drone based on physical guidance according to claim 1, characterized in that: The method of adaptive image motion compensation in step S3 is: S301, calculate compensation error: ， In the above formula, is the predicted compensation error, which is used to adjust the transformation matrix of the next frame. , is the weight coefficient, Used to balance the first-order effects, Used to balance the impact of second-order differences, , It is obtained through ridge regression training, and its value is in the range of [-10,10]. is the first-order difference, is the difference in transformation matrix between the current frame and the previous frame, is the second-order difference, and is the change in the first-order difference; ， In the above formula, is the final transformation matrix of the current frame, , is the final transformation matrix of the previous frame; S302, calculate the optimized transformation matrix: ， In the above formula, is the final transformation matrix at time t, is the predicted compensation error, It is an optimized transformation matrix used to reduce frame skipping and improve stability.

5. The method for enhancing the image stability of a UAV based on physical guidance according to claim 1, characterized in that: The method for lightweight machine learning optimization in S4 is: S401, is the optimized 2x3 affine transformation matrix, which is as follows: ， Specifically: ， ， In the above formula, are linear transformation parameters, which are used to control rotation, reduction, enlargement, and shear transformations. , is the translation parameter, Used to control the translation of the image in the x direction. Used to control the translation amount of the image in the y direction; S402, Boundary Correction: Let the original image be , then the image stabilization process based on the affine transformation function T is expressed as: ， The corrected result is: ， In the above formula, is the filled image, is the stabilized image, is a boundary mask used to identify the area that needs to be filled. It is a Laplace smoothing operator used to detect edges; Based on adaptive edge detection and local contrast enhancement, it is defined as follows: ， In the above formula, is the modulus of the image gradient, which can be calculated by the Sobel operator. is the mean of the local gradient, is the standard deviation of the gradient, is the Sigmoid function, which is used to Normalized to between 0 and 1; S403, calculating the final optimized image: ， In the above formula, For the final optimized image, is the predicted compensation error, The filled image.

Images