A method for enhancing the stability of UAV images based on physics guidance

Abstract

The present invention discloses a method for enhancing the stability of drone images based on physical guidance, which relates to the field of image enhancement technology and includes IMU sensor data fusion, physically-guided motion estimation, adaptive image motion compensation, and lightweight machine learning optimization. First, the present invention uses IMU data to correct the attitude drift error, and then combines the visual transformation matrix to calculate the global image stabilization transformation to achieve more accurate jitter compensation. Finally, through adaptive image compensation and lightweight machine learning optimization, the sense of frame skipping and boundary loss are reduced, and the image stabilization effect and calculation efficiency are improved. The present invention combines IMU sensor data, physical motion modeling, and adaptive image compensation to effectively reduce the impact of drone flight jitter on images and improve the image stabilization accuracy and calculation efficiency. Compared with traditional methods, this method enhances the physical consistency of image stabilization, reduces the sense of frame skipping and edge cropping loss. The present invention is applicable to high-dynamic drone aerial photography, mapping, and reconnaissance tasks in complex environments.

Description

Technical Field

[0001] The present invention relates to the technical field of image enhancement, and particularly to a method for enhancing the stability of UAV images based on physical guidance. Background Art

[0002] When a UAV flies in a complex environment, it is affected by airflow disturbance, attitude change, and control error, resulting in image jitter, which affects target recognition and mapping accuracy.

[0003] Currently, the methods for enhancing the stability of UAV images are mainly divided into two categories: mechanical image stabilization and electronic image stabilization. Mechanical image stabilization relies on a gimbal structure, and compensates for the attitude change of the UAV through gyroscopes and motor drives. It is suitable for low-speed and stable flight scenarios, but has limited response speed in high-dynamic flights. Electronic image stabilization estimates the image motion trajectory through visual algorithms and compensates for jitter using filtering and transformation, which has strong flexibility but is easily affected by cumulative errors. Traditional image stabilization methods include optical flow method, feature matching method, Kalman filtering, and L1 trajectory optimization, etc., but still have problems of insufficient robustness in the face of complex environments. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the above-mentioned disadvantages of the prior art and provide a method for enhancing the stability of UAV images based on physical guidance, which can effectively reduce the influence of UAV flight jitter on images, improve the image stabilization accuracy and calculation efficiency.

[0005] The technical solution adopted to solve the above technical problem is: a method for enhancing the stability of UAV images based on physical guidance, including the following steps: S1, IMU sensor data fusion: obtaining the attitude and displacement information of the UAV through IMU data fusion; S2, physically-guided motion estimation: performing spatial transformation of the image through a rotation matrix; S3, adaptive image motion compensation: optimizing video stability through adaptive image motion compensation; S4, lightweight machine learning optimization: avoiding edge cropping and enhancing edge details through lightweight machine learning optimization.

[0006] Further, the method for obtaining the attitude and displacement information of the UAV through IMU data fusion in S1 is as follows:

[0007] S101, collecting IMU data: the accelerometer measures the linear acceleration , , , is the linear acceleration along the x-axis, is the linear acceleration along the y-axis, is the linear acceleration along the z-axis, and the gyroscope measures the angular velocity , , , is the angular velocity about the x-axis, is the angular velocity about the y-axis, is the angular velocity about the z-axis;

[0008] S102, calculate the rotation angle:

[0009] ,

[0010] In the above formula, is the rotation angle at the current moment, is the rotation angle at the previous moment, is the angular velocity at the current moment and , is the time interval, is the drift error at time t and is corrected by adaptive filtering;

[0011] ,

[0012] In the above formula, is the adaptive weight, is the rotation angle calculated by the gyroscope, is the rotation angle calculated by the accelerometer, is the drift error at time t - 1;

[0013] ,

[0014] In the above formula, is the standard deviation of the gyroscope data, is the standard deviation of the accelerometer data;

[0015] S103, calculate the displacement of the UAV:

[0016] ,

[0017] In the above formula, is the position of the UAV at the current moment t, is the position of the UAV at the previous moment t - 1, is the velocity of the UAV at the previous moment t - 1, is the acceleration of the UAV at the current moment t and , is the time interval, is the compensation term;

[0018] ,

[0019] In the above formula, is the weight coefficient, is the past time index, is the decay factor, is the acceleration of the k-th frame, is the average acceleration of the past N frames, is the base of the natural logarithm.

[0020] Furthermore, the method of physically guided motion estimation in step S2 is as follows:

[0021] S201, calculate the rotation transformation matrix of the UAV camera, and this rotation matrix is the rotation state of the camera at time t:

[0022] ,

[0023] In the above formula, is the rotation transformation matrix of the UAV camera at time t, is the matrix rotating around the z-axis, is the rotation angle around the Z-axis, , is the matrix rotating around the y-axis, is the rotation angle around the Y-axis, , is the matrix rotating around the x-axis, is the rotation angle around the X-axis, ;

[0024] ,

[0025] S202, calculate the final motion transformation matrix:

[0026] ,

[0027] In the above formula, is the final transformation matrix at time t, is the visual transformation matrix.

[0028] Furthermore, the method of adaptive image motion compensation in step S3 is as follows:

[0029] S301, calculate the compensation error:

[0030] ,

[0031] In the above formula, is the predicted compensation error at time, which is used to adjust the transformation matrix of the next frame, , is the weight coefficient, which is used to balance the influence of the first-order difference, is used to balance the influence of the second-order difference, , Obtained through ridge regression training, and its value ranges within [-10, 10]. is the first-order difference, which is the difference in the transformation matrix between the current frame and the previous frame. is the second-order difference, which is the change amount of the first-order difference;

[0032] ,

[0033] In the above formula, is the final transformation matrix of the current frame, , is the final transformation matrix of the previous frame;

[0034] S302, calculate the optimized transformation matrix:

[0035] ,

[0036] In the above formula, is the final transformation matrix at time t, is the optimized 2x3 affine transformation matrix, which is used to reduce the sense of frame skipping and improve stability.

[0037] Furthermore, the method of lightweight machine learning optimization in the above S4 is as follows:

[0038] S401, is the optimized 2x3 affine transformation matrix, and its form is as follows:

[0039] ,

[0040] Specifically:

[0041] ,

[0042] ,

[0043] In the above formula, are linear transformation parameters, which are respectively used to control rotation, shrinking, magnification, and shear transformation, , are translation parameters, is used to control the translation amount of the image in the x direction, is used to control the translation amount of the image in the y direction;

[0044] S402, boundary correction:

[0045] Assume the original image is , then the image stabilization process based on the affine transformation function T is expressed as:

[0046] ,

[0047] The corrected result is as follows:

[0048] ,

[0049] In the above formula, is the filled image, is the stabilized image, is the boundary mask, used to identify the area that needs to be filled, is the Laplacian smoothing operator, used to detect edges;

[0050] Based on adaptive edge detection and local contrast enhancement, it is defined as follows:

[0051] ,

[0052] In the above formula, is the magnitude of the image gradient, which can be calculated by the Sobel operator, is the mean of the local gradient, is the standard deviation of the gradient, is the Sigmoid function, used to normalize it to between 0 and 1;

[0053] S403, calculate the final optimized image:

[0054] ,

[0055] In the above formula, is the finally optimized image, is the predicted compensation error at time t, is the filled image.

[0056] The beneficial effects of the present invention are as follows: (1) The present invention first corrects the attitude drift error using IMU data, and then calculates the global image stabilization transformation in combination with the visual transformation matrix to achieve more accurate jitter compensation. Finally, through adaptive image compensation and lightweight machine learning optimization, the sense of frame skipping and boundary loss are reduced, and the image stabilization effect and calculation efficiency are improved.

[0057] (2) The present invention combines IMU sensor data, physical motion modeling and adaptive image compensation to effectively reduce the impact of UAV flight jitter on the image, and improve the image stabilization accuracy and calculation efficiency. Compared with traditional methods, this method enhances the physical consistency of image stabilization, reduces the sense of frame skipping and edge cropping loss. The present invention is applicable to high-dynamic UAV aerial photography, mapping and reconnaissance tasks in complex environments.

[0058] (3) The present invention estimates the attitude and displacement of the UAV based on IMU data, optimizes the image stabilization calculation by combining physical motion modeling, corrects the gyroscope drift error using adaptive filtering, and introduces a displacement compensation term to correct the acceleration accumulation error, so as to improve the short-term motion estimation accuracy.

[0059] (4) The present invention combines the rotation matrix and the visual transformation matrix to calculate the global motion transformation matrix, so as to optimize the UAV camera jitter compensation. Through the error compensation strategy trained by ridge regression, the first-order difference and the second-order difference are calculated to predict the image stabilization error, and the transformation matrix is optimized to reduce the sense of frame skipping.

[0060] (5) The present invention identifies the cropping area of the image stabilization image based on the adaptive boundary mask of the Sobel operator, and performs boundary repair through Laplacian smoothing to reduce information loss. Finally, the optimized image is calculated by combining the image stabilization image, error compensation and filling area, ensuring the integrity of the image after image stabilization and the retention of edge details. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a curve graph of the PSNR performance of the present invention in the original image, motion compensation, deblurring and fully optimized algorithm steps.

[0062] Figure 2 It is a curve graph of the SSIM performance in the original image, motion compensation, deblurring and fully optimized algorithm steps.

[0063] Figure 3 It is a curve graph of the FPS in the original image, motion compensation, deblurring and fully optimized algorithm steps. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0064] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0065] A method for enhancing the stability of UAV images based on physical guidance in this embodiment includes the following steps:

[0066] S1, IMU sensor data fusion: Obtain the attitude and displacement information of the UAV through IMU data fusion, and these data provide necessary inputs for physically guided motion estimation.

[0067] Obtain the attitude and displacement information of the UAV through IMU data fusion to reduce the influence of sensor errors on image stabilization. Calculate the rotation angle using gyroscope integration, and correct the drift error through adaptive filtering. Use the accelerometer for secondary integration to calculate the displacement, and introduce an innovative displacement compensation term To optimize the short-term jitter error. By combining gyroscope and accelerometer data, the accuracy of the UAV camera motion estimation is improved, providing reliable physical prior information for subsequent image stabilization compensation.

[0068] IMU (Inertial Measurement Unit)-physically guided UAV image stability enhancement provides precise physical data to adjust the camera attitude by real-time sensing of the UAV's attitude and motion state, effectively reducing jitter and vibration, being able to quickly respond to dynamic changes during flight, and ensuring the stability of images and videos. IMU-guided stability enhancement does not rely on post-processing, directly improving the real-time shooting quality and being applicable to complex flight environments.

[0069] The method for obtaining the UAV's attitude and displacement information through IMU data fusion is as follows:

[0070] S101, collect IMU data: The accelerometer measures the linear acceleration , , , is the linear acceleration along the x-axis, is the linear acceleration along the y-axis, is the linear acceleration along the z-axis, and the gyroscope measures the angular velocity , , , is the angular velocity around the x-axis, is the angular velocity around the y-axis, is the angular velocity around the z-axis;

[0071] S102, calculate the rotation angle:

[0072] ,

[0073] In the above formula, is the rotation angle at the current moment, is the rotation angle at the previous moment, is the angular velocity at the current moment and , is the time interval, is the drift error at time t, which is corrected through adaptive filtering;

[0074] ,

[0075] In the above formula, is the adaptive weight, is the rotation angle calculated by the gyroscope, is the rotation angle calculated by the accelerometer, is the drift error at time t-1;

[0076] ,

[0077] In the above formula, is the standard deviation of the gyroscope data, is the standard deviation of the accelerometer data;

[0078] S103. Calculate the displacement of the UAV:

[0079] ,

[0080] In the above formula, is the position of the UAV at the current time t, is the position of the UAV at the previous time t - 1, is the velocity of the UAV at the previous time t - 1, is the acceleration of the UAV at the current time t and , is the time interval, is the compensation term;

[0081] ,

[0082] In the above formula, is the weight coefficient, is the past time index, is the decay factor, is the acceleration of the k-th frame, is the average acceleration of the past N frames, is the base of the natural logarithm.

[0083] S2. Physically-guided motion estimation: Perform spatial transformation of the image through the rotation matrix, and the estimation result is used for adaptive image motion compensation for error correction.

[0084] Perform spatial transformation of the image through the rotation matrix to establish an accurate motion model. Calculate the rotation matrices for rotation around the x-axis, y-axis, and z-axis, and obtain the total rotation transformation matrix through the product of the above matrices, which is used to transform the image from the camera coordinate system to the world coordinate system. By multiplying this rotation transformation matrix with the visual transformation matrix, calculate the final image transformation matrix for the final correction of the image.

[0085] The method of physically-guided motion estimation is:

[0086] S201. Calculate the rotation transformation matrix of the UAV camera, and this rotation matrix is the rotation state of the camera at time t:

[0087] ,

[0088] In the above formula, is the rotation transformation matrix of the UAV camera at time t, is the matrix rotated around the z-axis, is the rotation angle around the Z-axis, , is the matrix rotated around the y-axis, is the rotation angle around the Y-axis, , is the matrix rotated around the x-axis, is the rotation angle around the X-axis, ;

[0089] ,

[0090] S202, calculate the final motion transformation matrix:

[0091] ,

[0092] In the above formula, is the final transformation matrix at time t, is the visual transformation matrix.

[0093] S3, Adaptive Image Motion Compensation: Optimize video stability through adaptive image motion compensation. To further optimize stability, lightweight machine learning optimization is introduced for fine-tuning.

[0094] Optimize video stability through adaptive image motion compensation. Calculate the compensation error, combining the transformation matrix difference and second-order difference of the previous frame. These parameters are trained by ridge regression to minimize the error and prevent overfitting. Use the calculated error to optimize the transformation matrix of the current frame, thereby reducing the sense of frame skipping and improving the stability of the video, effectively enhancing the smoothness of image processing and visual experience.

[0095] The method of adaptive image motion compensation is:

[0096] S301, calculate the compensation error:

[0097] ,

[0098] In the above formula, is the predicted compensation error at time, used to adjust the transformation matrix of the next frame, , is the weight coefficient, used to balance the influence of the first-order difference, used to balance the influence of the second-order difference, , obtained by ridge regression training, and its value is within the range of [-10, 10], is the first-order difference, which is the difference between the transformation matrices of the current frame and the previous frame, is the second-order difference, which is the change amount of the first-order difference;

[0099] ,

[0100] In the above formula, is the final transformation matrix of the current frame, , is the final transformation matrix of the previous frame;

[0101] S302, calculate the optimized transformation matrix:

[0102] ,

[0103] In the above formula, is the final transformation matrix at time t, is the optimized 2x3 affine transformation matrix, which is used to reduce the sense of frame skipping and improve stability.

[0104] S4, lightweight machine learning optimization: Avoid edge clipping and enhance edge details through lightweight machine learning optimization, correct edge information through boundary filling and Laplacian smoothing, avoid clipping, and improve visual quality.

[0105] Avoid edge clipping and enhance edge details through lightweight machine learning optimization. Perform boundary correction, and use boundary masks and Laplacian smoothing operations to fill the edge regions of the stable image. Calculate the final optimized image, combining the stable image, the predicted compensation error, and the filled image to ensure the retention of edge details and the integrity of the image.

[0106] The method of lightweight machine learning optimization is:

[0107] S401, is the optimized 2x3 affine transformation matrix, and its form is as follows:

[0108] ,

[0109] Specifically:

[0110] ,

[0111] ,

[0112] In the above formula, are the linear transformation parameters, which are used to control rotation, reduction, enlargement, and shear transformation respectively, , are the translation parameters, is used to control the translation amount of the image in the x direction, is used to control the translation amount of the image in the y direction;

[0113] S402, Boundary correction:

[0114] Let the original image be , then the image stabilization process based on the affine transformation function T is expressed as:

[0115] ,

[0116] Then the corrected result is:

[0117] ,

[0118] In the above formula, is the filled image, is the stabilized image, is the boundary mask, used to identify the area that needs to be filled, is the Laplacian smoothing operator, used to detect edges;

[0119] Based on adaptive edge detection and local contrast enhancement, it is defined as follows:

[0120] ,

[0121] In the above formula, is the magnitude of the image gradient, which can be calculated by the Sobel operator, is the mean of the local gradient, is the standard deviation of the gradient, is the Sigmoid function, used to normalize to between 0 and 1.

[0122] S403, Calculate the final optimized image:

[0123] ,

[0124] In the above formula, is the finally optimized image, is the predicted compensation error at time t, is the filled image.

[0125] The quantitative analysis of the image stabilization effect of this embodiment uses PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity Index), and FPS (Frames Per Second). The experimental results are as Figures 1 to 3 shown. The algorithm optimization significantly improves the image quality. The improvement of PSNR and SSIM indicates that the image quality is gradually improved. During the process of enhancing stability, the change in the frame rate is small, ensuring the real-time performance of the algorithm.

[0126] The above is only a preferred embodiment of the present invention and is not used to limit the protection scope of the present invention.

Claims

1. A method for enhancing the image stability of a drone based on physical guidance, characterized in that: The following steps are involved: S1, IMU sensor data fusion: obtain the attitude and displacement information of the drone through IMU data fusion; S2, physics-guided motion estimation: spatial transformation of the image via a rotation matrix; S3, Adaptive Image Motion Compensation: Optimize video stability through adaptive image motion compensation; The method of adaptive image motion compensation in step S3 is: S301, calculate compensation error: ， In the above formula, for The compensation error of the moment prediction is used to adjust the transformation matrix of the next frame. , is the weight coefficient, Used to balance the effect of first-order differences, Used to balance the impact of second-order differences, , It is obtained through ridge regression training, and its value is in the range of [-10,10]. is the first-order difference, is the difference in transformation matrix between the current frame and the previous frame, is the second-order difference, and is the change in the first-order difference; ， In the above formula, is the final transformation matrix of the current frame, , is the final transformation matrix of the previous frame; S302, calculate the optimized transformation matrix: ， In the above formula, is the final transformation matrix at time t, It is an optimized 2x3 affine transformation matrix, which is used to reduce frame skipping and improve stability; S4, lightweight machine learning optimization: avoid edge clipping and enhance edge details through lightweight machine learning optimization; The method for lightweight machine learning optimization in S4 is: S401, is the optimized 2x3 affine transformation matrix, which is as follows: ， Specifically: ， In the above formula, are linear transformation parameters, which are used to control rotation, reduction, enlargement, and shear transformations. , is the translation parameter, Used to control the translation of the image in the x direction. Used to control the translation amount of the image in the y direction; S402, Boundary Correction: Let the original image be , then the image stabilization process based on the affine transformation function T is expressed as: ， The corrected result is: ， In the above formula, is the filled image, is the stabilized image, is a boundary mask used to identify the area that needs to be filled. It is a Laplace smoothing operator used to detect edges; Based on adaptive edge detection and local contrast enhancement, it is defined as follows: ， In the above formula, is the modulus of the image gradient, which can be calculated by the Sobel operator. is the mean of the local gradient, is the standard deviation of the gradient, is the Sigmoid function, which is used to Normalized to between 0 and 1; S403, calculating the final optimized image: ， In the above formula, For the final optimized image, is the compensation error predicted at time t, The filled image.

2. The method for enhancing the image stability of a UAV based on physical guidance according to claim 1, characterized in that: The method for obtaining the attitude and displacement information of the drone through IMU data fusion in S1 is: S101, collect IMU data: accelerometer measures linear acceleration , , , is the linear acceleration along the x-axis, is the linear acceleration along the y-axis, is the linear acceleration along the z-axis, and the gyroscope measures the angular velocity , , , is the angular velocity around the x-axis, is the angular velocity around the y-axis, is the angular velocity around the z axis; S102, calculate the rotation angle: , In the above formula, is the rotation angle at the current moment, is the rotation angle at the previous moment, is the angular velocity at the current moment and , is the time interval, is the drift error at time t, which is corrected by adaptive filtering; ， In the above formula, is the adaptive weight, is the rotation angle calculated by the gyroscope, is the rotation angle calculated by the accelerometer, is the drift error at time t-1; ， In the above formula, is the standard deviation of the gyroscope data, is the standard deviation of the accelerometer data; S103, calculate the displacement of the drone: ， In the above formula, is the position of the UAV at the current time t, is the position of the drone at the last moment t-1, is the speed of the drone at the last moment t-1, is the acceleration of the drone at the current time t and , is the time interval, For compensation items; ， In the above formula, is the weight coefficient, is the past time index, is the attenuation factor, is the acceleration of the kth frame, is the average acceleration of the past N frames, is the base of natural logarithms.

3. The method for enhancing the image stability of a UAV based on physical guidance according to claim 2, characterized in that: The method of physically guided motion estimation in step S2 is: S201, calculate the rotation transformation matrix of the drone camera, which is the rotation state of the camera at time t: ， In the above formula, is the rotation transformation matrix of the drone camera at time t, is the matrix for rotation around the z-axis, is the rotation angle around the Z axis, , is the matrix for rotation around the y axis, is the rotation angle around the Y axis, , is the matrix for rotation around the x-axis, is the rotation angle around the X axis, ; ， S202, calculate the final motion transformation matrix: ， In the above formula, is the final transformation matrix at time t, is the visual transformation matrix.

Images