An EIS-based lightweight electronic anti-shake method and system

Abstract

The application relates to the technical field of electronic anti-shake, and particularly discloses a lightweight electronic anti-shake method and system based on EIS, which comprises the following steps: constructing and adopting a double-path data acquisition pipeline to obtain image frame data flow and inertial data flow; performing real-time interframe basic motion estimation and absolute attitude calculation based on the inertial data flow; separately solving relative three-dimensional rotation change and absolute attitude angle matrix and fusing the same; constructing a composite affine transformation matrix; combining the image frame data flow and the composite affine transformation matrix; judging whether visual auxiliary data fusion is triggered in real time; generating an optimal affine transformation matrix; recording an IMU dominant duration when the visual auxiliary data fusion is not triggered; judging whether motion prediction correction updating of the optimal affine transformation matrix is triggered according to the IMU dominant duration; performing motion smoothing processing to output a stable affine transformation matrix; performing geometric transformation on the image frame based on the stable affine transformation matrix; and generating a stable image frame.

Description

Technical Field

[0001] This invention relates to the field of electronic image stabilization technology, specifically a lightweight electronic image stabilization method and system based on EIS. Background Technology

[0002] In automated operations of agricultural unmanned equipment, such as plant protection, fertilization, and growth monitoring, the stability of video captured by airborne cameras is crucial, forming the basis for subsequent precision agriculture analysis and decision-making. However, in practice, it has been found that the continuous high-frequency mechanical vibrations introduced by the rotation of the propeller and the operation of the engine, coupled with environmental factors such as wind disturbance during aerial operations, result in severe shaking in the captured video footage, seriously affecting the usability of the video and the accuracy of subsequent analysis. This problem was discovered directly from the observation and analysis of actual field operation videos.

[0003] To eliminate shaking, electronic image stabilization technology has emerged. However, after in-depth analysis and technical verification for the specific scenario of agricultural unmanned equipment, we found that existing technical solutions have several technical problems that urgently need to be solved:

[0004] First, the limitations of single-sensor solutions are significant. Pure vision methods estimate motion by analyzing image sequences, but in areas with uniform textures that agricultural equipment frequently flies over, such as uniform crop canopies or bare fields, the lack of image features can lead to motion estimation failures, causing image stuttering or jumps. While pure IMU solutions have high-frequency responses, their gyroscope data accumulates drift errors when integrating to solve for attitude, causing slow, irreversible slippage in the image after long-term operation, making it impossible to maintain long-term stability. On the other hand, traditional IMU and vision fusion strategies are not robust enough in complex agricultural scenarios. Existing solutions often assume that visual data is always reliable or use fixed fusion weights. However, in actual farmland environments, sudden changes in lighting and large areas of repetitive textures occur frequently, making visual motion estimation prone to mismatches. If unreliable visual results are fused with IMU data without discrimination, errors will be introduced, degrading the image stabilization effect and even exacerbating image jitter.

[0005] To address the above problems, this invention proposes a lightweight electronic image stabilization method and system based on EIS. Summary of the Invention

[0006] The purpose of this invention is to provide a lightweight electronic image stabilization method and system based on EIS to solve the aforementioned background problems.

[0007] The objective of this invention can be achieved through the following technical solution: a lightweight electronic image stabilization method based on EIS, comprising:

[0008] A dual-channel data acquisition pipeline was constructed and used to acquire image frame data streams and inertial data streams;

[0009] Based on inertial data stream, real-time inter-frame basic motion estimation and absolute attitude calculation are performed. The relative three-dimensional rotation transformation and absolute attitude angle matrices are solved separately and then fused to construct a composite affine transformation matrix.

[0010] By combining the image frame data stream with the composite affine transformation matrix, it is determined in real time whether visual auxiliary data fusion is triggered, the optimal affine transformation matrix is ​​generated, and the duration of IMU dominance is recorded when it is not triggered.

[0011] Based on the duration of IMU dominance, determine whether to trigger the motion prediction correction update of the optimal affine transformation matrix, and perform motion smoothing to output a stable affine transformation matrix;

[0012] A geometric transformation is performed on the image frame based on the stable affine transformation matrix to generate a stable image frame;

[0013] Furthermore, the method for obtaining the relative three-dimensional rotational changes is as follows:

[0014] The inertial data stream includes gyroscope data and accelerometer data. The gyroscope data in the inertial data stream is acquired within the time interval from the previous image frame to the current image frame to obtain the inter-frame gyroscope data sequence. Based on the inter-frame gyroscope data sequence, the relative three-dimensional rotation change is obtained by constructing a quaternion differential equation and performing numerical integration using the fourth-order Runge-Kutta method. The solution is recursively obtained in the form of attitude quaternions.

[0015] Furthermore, the absolute attitude angle matrix is ​​obtained as follows:

[0016] Based on the gyroscope and accelerometer data included in the inertial data stream, an absolute attitude calculation is performed using a complementary filter fusion algorithm. Low-frequency attitude signals of pitch and roll angles are obtained by calculating the pitch and roll angles using the accelerometer data. Based on the continuous gyroscope data, the high-frequency attitude signals are obtained by integrating and solving quaternion differential equations and converting them into Euler angles. The low-frequency attitude signals and high-frequency attitude signals are then weighted and fused using a complementary filter to output an absolute attitude angle matrix including pitch, yaw, and roll angles.

[0017] Furthermore, the composite affine transformation matrix is ​​constructed as follows:

[0018] The relative three-dimensional rotation transformation is decomposed and projected onto the two-dimensional image plane to obtain the basic affine transformation used to compensate for inter-frame jitter. Based on the absolute attitude angle matrix, a rotation transformation is generated to actively correct the image tilt and maintain the horizon level. The basic affine transformation and the rotation transformation are combined by matrix multiplication to obtain the composite affine transformation matrix.

[0019] Furthermore, the optimal affine transformation matrix is ​​generated as follows:

[0020] The texture richness of the current image frame is evaluated. If the texture richness is high, fast optical flow calculation is performed on the current image frame and the previous stable image frame to fit the image affine transformation matrix. The stable image frame represents the image frame output after the corresponding image frame has undergone the entire electronic image stabilization process. The Euclidean distance between the composite affine transformation matrix and the image affine transformation matrix in the vector space is calculated. If the Euclidean distance is less than the preset fault tolerance threshold, the IMU dominance duration is set to zero, triggering visual auxiliary data fusion based on Kalman filtering. The composite affine transformation matrix and the image affine transformation matrix are fused to generate the optimal affine transformation matrix.

[0021] If the Euclidean distance is greater than or equal to the fault tolerance threshold, or the current texture richness is low, the IMU-dominated duration starts the timing, and the composite affine transformation matrix is ​​used as the optimal affine transformation matrix output.

[0022] Furthermore, the texture richness is evaluated as follows:

[0023] The Laplacian operator is used to perform convolution operation on the current image frame, and the Laplacian variance of the output image is calculated as a quantitative indicator of image texture richness. If the Laplacian variance is lower than the preset variance threshold, the current texture richness is judged to be low; otherwise, the current texture richness is judged to be high.

[0024] Furthermore, the motion prediction correction and update method is as follows:

[0025] Maintain a fixed-length first-in-first-out queue, store the optimal affine transformation matrix sequence, read the three-dimensional flight velocity vector of the unmanned equipment in real time from the flight control system for the corresponding time period, organize it to obtain the flight velocity sequence, obtain the absolute attitude angle matrix of the unmanned equipment for the corresponding time period, and organize it to obtain the body attitude sequence.

[0026] Time series regression analysis is performed based on the optimal affine transformation matrix sequence to predict the predicted increment changes of each component of the optimal affine transformation matrix. Combined with the flight speed sequence and the aircraft attitude sequence, the actual increment changes are estimated through kinematic prediction. Deviation analysis is performed between the predicted increment changes and the actual increment changes to obtain the baseline drift trend and perform geometric reversal to obtain the inverse transformation. The inverse transformation is then geometrically synthesized and corrected with the optimal affine transformation matrix of the current image frame to complete the update.

[0027] Furthermore, the stable affine transformation matrix is ​​obtained as follows:

[0028] An exponentially weighted moving average filter is used as the motion filtering algorithm to perform motion smoothing on the optimal affine transformation matrix or the optimal affine transformation matrix updated by motion prediction. The low-frequency components representing the intentional flight path of the unmanned equipment are retained, while the high-frequency jitter components representing high-frequency mechanical vibration and random jitter are attenuated, and the stable affine transformation matrix after motion smoothing is output.

[0029] Furthermore, the stable image frame is generated as follows:

[0030] Using the stable affine transformation matrix as the affine transformation parameter, the current image frame is transformed pixel by pixel using the bilinear interpolation algorithm to generate an intermediate image frame. The intermediate image frame after geometric transformation is adaptively cropped to identify the effective content region and calculate the maximum inscribed rectangle. The Lanczos resampling algorithm is then used to scale the cropped stable image to the preset output resolution to generate a stable image frame.

[0031] A lightweight electronic image stabilization system based on EIS includes the following modules:

[0032] Acquisition module: Constructs and employs a dual-channel data acquisition pipeline to acquire image frame data streams and inertial data streams;

[0033] IMU solution module: Based on inertial data stream, it performs inter-frame basic motion estimation and absolute attitude solution in real time, solves for relative three-dimensional rotation transformation and absolute attitude angle matrix respectively, and fuses them to construct composite affine transformation matrix;

[0034] Visual assistance module: Combines image frame data stream with composite affine transformation matrix to determine in real time whether visual assistance data fusion is triggered, generates the optimal affine transformation matrix, and records the IMU dominance duration when not triggered;

[0035] Motion Correction and Smoothing Module: Determines whether to trigger motion prediction correction and update of the optimal affine transformation matrix based on the IMU dominance duration, and performs motion smoothing to output a stable affine transformation matrix;

[0036] Image processing module: Performs geometric transformations on image frames based on a stable affine transformation matrix to generate stable image frames.

[0037] The beneficial effects of this invention are as follows:

[0038] 1. This invention introduces a dual criterion based on image texture richness assessment and motion estimation consistency test to intelligently decide whether to enable visual-assisted data fusion. This allows the system to utilize the absolute reference advantage of visual information in farmland areas with rich texture features, and output high-precision motion estimation through Kalman filtering fusion, effectively suppressing high-frequency jitter. When flying over a uniform canopy with a single color, the system can automatically downgrade to pure IMU mode to avoid the risk of visual mismatch, ensuring the continuity and reliability of anti-shake processing in complex agricultural scenarios, and fundamentally enhancing the system's environmental adaptability.

[0039] 2. This invention effectively ensures the long-term stability and engineering feasibility of image stabilization by introducing time-based drift compensation and lightweight algorithm design. The solution monitors the duration of IMU dominance to warn of long-term visual information loss and triggers a motion prediction correction mechanism. It uses the aircraft kinematic model to compensate for the cumulative drift of the IMU, thereby solving the problem of slow image slippage inherent in pure inertial navigation. At the same time, the entire chain adopts computationally efficient algorithms such as complementary filtering and EWMA smoothing. While ensuring core performance, it handles geometric transformation boundaries through adaptive cropping and scaling. Finally, it achieves professional-grade, stable video output without cumulative error on the limited computing resources of agricultural equipment. Attached Figure Description

[0040] The invention will now be further described with reference to the accompanying drawings.

[0041] Figure 1 This is a flowchart illustrating the steps of a lightweight electronic image stabilization method based on EIS as described in an embodiment of the present invention.

[0042] Figure 2 This is a module architecture diagram of a lightweight electronic image stabilization system based on EIS as described in an embodiment of the present invention. Detailed Implementation

[0043] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0044] Example 1

[0045] Please see Figure 1 As shown in the embodiment of the present invention, a lightweight electronic image stabilization method based on EIS aims to solve the problems of video jitter caused by high-frequency vibration of the machine body, easy failure of visual assistance due to texture loss, and cumulative drift of inertial measurement unit (IMU) calculation in complex operating scenarios, which affect the stability of the image. The method solves the relative three-dimensional rotation change and absolute attitude angle matrix of the machine body separately and fuses them into a composite affine transformation matrix. Then, it combines image texture richness evaluation and optical flow calculation to adaptively trigger visual assistance data fusion based on Kalman filtering to generate the optimal affine transformation matrix. Furthermore, it introduces a motion prediction correction mechanism based on IMU-dominated duration to compensate for the reference drift under long-term visual loss. Finally, it performs motion smoothing and adaptive cropping to achieve a high-quality electronic image stabilization effect, including the following steps:

[0046] S1: Construct and employ a dual-channel data acquisition pipeline to acquire image frame data streams and inertial data streams;

[0047] Specifically, based on the processing requirements of electronic image stabilization (EIS) for agricultural unmanned equipment, an image sensor and an inertial measurement unit (IMU) are integrated into the flight control system of the agricultural unmanned equipment. A dual data acquisition pipeline is constructed, including an image sensor pipeline and an IMU data pipeline. The image sensor pipeline is triggered and captures image frames at a preset frame rate based on the video stream interface to obtain the image frame data stream. The IMU data pipeline is asynchronously acquired and outputs gyroscope data and accelerometer data at a sampling frequency significantly higher than the preset frame rate based on the built-in clock of the inertial measurement unit, forming a high-throughput raw inertial data stream.

[0048] Among them, the gyroscope data represents the body's rotational angular velocity obtained based on the gyroscope in the inertial measurement unit, and its function is to track rotational motion; the accelerometer data represents the body's linear acceleration obtained based on the accelerometer in the inertial measurement unit, and its function is to track linear motion.

[0049] It should be noted that EIS is a technology system that uses software algorithms and sensor data to perform real-time geometric transformations on image frames to compensate for carrier vibration. Its purpose is to eliminate image jitter caused by machine vibration during the operation of agricultural unmanned equipment, and improve the stability and usability of the acquired video.

[0050] The raw inertial data stream is preprocessed. Specifically, a first-order infinite impulse response (IIR) low-pass filter is used to filter the gyroscope and accelerometer data in real time. The cutoff frequency is set lower than the fundamental frequency and main harmonic frequency of the propeller rotation of the agricultural unmanned equipment. High-frequency mechanical vibration noise introduced by the aerodynamic effects of the propeller and the operation of the engine is filtered out, while the low-frequency inertial signal that characterizes the autonomous motion of the equipment is retained. After preprocessing, an effective inertial data stream is obtained.

[0051] It should be noted that the purpose of this step is to acquire image frame data streams and raw inertial data streams through the constructed dual-channel data acquisition pipeline, and to perform targeted low-pass filtering preprocessing on the raw inertial data stream. This effectively suppresses high-frequency vibration noise at the data source, providing a high-quality, low-noise inertial data stream foundation for subsequent accurate motion estimation and attitude calculation, and laying the signal quality foundation for the entire chain of anti-shake processing.

[0052] S2: Based on the inertial data stream, perform real-time inter-frame basic motion estimation and absolute attitude calculation, solve for the relative three-dimensional rotation transformation and absolute attitude angle matrices respectively, and fuse them to construct a composite affine transformation matrix;

[0053] The gyroscope data in the inertial data stream is obtained within the time interval from the previous image frame to the current image frame, and the inter-frame gyroscope data sequence is obtained. Based on the inter-frame gyroscope data sequence, the inter-frame basic motion estimation is performed to solve the relative three-dimensional rotation change.

[0054] Specifically, for inter-frame gyroscope data sequences, quaternion differential equations are constructed and numerical integration is performed on the quaternion differential equations using the fourth-order Runge-Kutta method. The relative three-dimensional rotation changes within the time interval of adjacent image frames are obtained by recursively solving the gyroscope data. The relative three-dimensional rotation changes are represented in the form of attitude quaternions, which quantifies the inter-frame angle changes of the unmanned equipment's body coordinate system on the pitch, yaw, and roll axes, directly corresponding to the high-frequency jitter between image frames.

[0055] It should be noted that the meaning of the quaternion differential equation is to describe the mathematical relationship between the body's rotational angular velocity and the time-varying rate of the attitude quaternion, and the body's rotational angular velocity is the gyroscope data.

[0056] Based on the gyroscope and accelerometer data included in the inertial data stream, a complementary filtering fusion algorithm is used to calculate the absolute attitude and solve for the absolute attitude angle matrix. The absolute attitude angle matrix includes the pitch angle, yaw angle and roll angle of the body coordinate system relative to the geographic coordinate system.

[0057] Specifically, using accelerometer data, the pitch and roll angles are calculated using the arctangent function by the ratio of gravitational acceleration components in the body coordinate system. Based on continuous gyroscope data from the start of the current operation of the unmanned equipment, a quaternion differential equation is constructed and the attitude quaternion is recursively solved by numerical integration using the fourth-order Runge-Kutta method. The result is converted to Euler angles to obtain high-frequency attitude signals including pitch, yaw, and roll angles. A complementary filter is designed to weight and fuse the calculated low-frequency and high-frequency attitude signals. By adjusting the cutoff frequency of the complementary filter, the low-frequency component is trusted to rely on the low-frequency attitude signal calculated from the accelerometer data to suppress the drift error accumulated by the integration of gyroscope data, while the high-frequency component is trusted to rely on the high-frequency attitude signal calculated from the gyroscope data to maintain dynamic response. A stable absolute attitude angle matrix is ​​output, including the pitch, yaw, and roll angles of the body coordinate system relative to the geographic coordinate system.

[0058] It should be noted that gyroscope data integration accumulates drift error, while accelerometers can provide a reliable reference under static or low-speed motion conditions.

[0059] By fusing the three-dimensional rotational transformation with the absolute attitude angle matrix, a composite affine transformation matrix is ​​constructed.

[0060] Specifically, the relative three-dimensional rotation transformation is decomposed and projected onto the two-dimensional image plane to obtain the basic affine transformation used to compensate for inter-frame jitter. Based on the absolute attitude angle matrix, a rotation transformation is generated to actively correct the image tilt and maintain the horizon level. The basic affine transformation and the rotation transformation are combined by matrix multiplication to obtain the composite affine transformation matrix H_imu.

[0061] It should be noted that the composite affine transformation matrix H_imu considers both high-frequency jitter compensation and static or near-static tilt correction.

[0062] S3: Combine image frame data stream with composite affine transformation matrix to determine in real time whether visual auxiliary data fusion is triggered, generate the optimal affine transformation matrix, and record the IMU dominance duration when not triggered;

[0063] Specifically, the image frame data stream is acquired, and the texture richness of the current image frame is evaluated. The Laplacian operator is used to perform convolution operation on the current image frame, and the Laplacian variance of the output image is calculated as a quantitative indicator of the texture richness of the image. If the Laplacian variance is lower than the preset variance threshold, the current texture richness is judged to be low; otherwise, the current texture richness is judged to be high, and visual auxiliary verification is triggered on the composite affine transformation matrix.

[0064] It should be noted that the higher the Laplacian variance of the current image frame, the richer the image edge and texture features, which is more conducive to the stable operation of the visual motion estimation algorithm. The role of texture richness evaluation is to build the adaptive switching capability between feature-rich scenes and feature-deficient scenes to determine whether visual auxiliary verification is activated.

[0065] For example, in farmland operation scenarios, when the drone flies over crop fields with rich textures and distinct furrows, the calculated Laplace variance value is higher, while when it flies over uniform crop canopy areas with a single color, the calculated Laplace variance value is significantly lower.

[0066] Fast optical flow calculation is performed on the current image frame and the previous stable image frame. The LK optical flow method is used. Strong corner point features in the image frame are selected as tracking pixels. The optical flow vector field in the neighborhood of the tracking pixel is solved iteratively, and the image affine transformation matrix H_vision based on the image sequence is obtained by fitting.

[0067] Among them, the stable image frame refers to the image frame output after the corresponding image frame has undergone the entire electronic image stabilization process;

[0068] Set an IMU-dominant duration and initialize it to 0;

[0069] Treat the composite affine transformation matrix H_imu and the image affine transformation matrix H_vision as vectors, calculate the Euclidean distance between the corresponding vector spaces, and compare the calculated Euclidean distance with a preset fault tolerance threshold to perform a consistency check on motion estimation.

[0070] If the Euclidean distance is less than the fault tolerance threshold, it is determined that the IMU and the visual motion estimation results are consistent, triggering visual-assisted data fusion. A Kalman filter is used as the data fusion algorithm, with H_imu as the predicted value and H_vision as the observed value. Through dynamic weighted fusion using Kalman gain, the optimal affine transformation matrix H_fused is output, and the IMU dominance duration is set to zero.

[0071] If the Euclidean distance is greater than or equal to the fault tolerance threshold, or the current texture richness is low, the composite affine transformation matrix H_imu is directly used as the output of the optimal affine transformation matrix H_fused, and the IMU-dominated duration is started to be counted.

[0072] It should be noted that the role of consistency check is to identify motion estimation anomalies caused by visual mismatches and ensure the reliability of the fusion results. Data fusion makes full use of the high-frequency response characteristics of IMU data and the absolute reference advantage of visual data, aiming to improve the overall accuracy and stability of motion estimation.

[0073] It should be noted that the purpose of this step is to build an adaptive, multi-sensor cross-validation data processing flow, which verifies and calibrates the reliability of the IMU base estimate through visual information, and performs optimal fusion when conditions permit, thereby significantly improving the image stabilization accuracy and visual quality in feature-rich scenarios while ensuring the basic robustness of the system.

[0074] S4: Determine whether to trigger the motion prediction correction update of the optimal affine transformation matrix based on the IMU dominance duration, and perform motion smoothing to output a stable affine transformation matrix;

[0075] Specifically, if the IMU-dominated duration exceeds the preset duration standard, it is determined that the visual features are continuously missing, triggering the motion prediction correction update of the optimal affine transformation matrix. A fixed-length first-in-first-out (FIFO) queue is maintained to store the optimal affine transformation matrix sequence corresponding to the image frame sequence ending with the previous image frame. The three-dimensional flight velocity vector of the unmanned vehicle in the corresponding time period is read from the flight control system in real time and processed to obtain the flight velocity sequence. The absolute attitude angle matrix of the unmanned vehicle in the corresponding time period is obtained and processed to obtain the body attitude sequence.

[0076] Independent time series regression analysis was performed on the component sequences of the optimal affine transformation matrix in the optimal affine transformation matrix sequence. The least squares method was used to fit the linear regression model, and the incremental changes of each component at the next frame time were predicted by extrapolation and marked as predicted incremental changes. Based on the flight velocity sequence and the aircraft attitude sequence, the incremental changes of each component generated by the autonomous flight intention were estimated by kinematic prediction and marked as actual incremental changes. The deviation analysis between the predicted incremental changes and the actual incremental changes was performed to obtain the baseline drift trend.

[0077] It should be noted that the linear regression model can extract the dominant and smooth motion trend, and the prediction and estimation results quantify the expected motion changes that the unmanned equipment will produce according to its current motion inertia if there are no new random jitter inputs.

[0078] The optimal affine transformation matrix of the current image frame is corrected and updated based on the predicted baseline drift trend. The predicted baseline drift trend is geometrically reversed to obtain the inverse transformation. The inverse transformation is geometrically synthesized and corrected with the optimal affine transformation matrix of the current image frame. The optimal affine transformation matrix is ​​updated using the result of the geometric synthesis correction.

[0079] It should be noted that the baseline drift trend represents the predicted drift trend. The purpose of geometric synthesis correction is to apply a force that is equal in magnitude and opposite in direction to the predicted drift trend, so that the slowly accumulating abnormal displacement or rotation that originally existed in the optimal affine transformation matrix returns to the expected trajectory defined by the kinematic prediction trend, thus compensating for the cumulative drift error that may be caused by IMU integration.

[0080] It should be noted that if motion prediction correction is not triggered, the optimal affine transformation matrix does not need to be corrected or updated.

[0081] An exponentially weighted moving average (EWMA) filter is used as a motion filtering algorithm to smooth the optimal affine transformation matrix. The EWMA filter retains the low-frequency components that represent the intentional flight path of the unmanned equipment based on a preset time constant, attenuates the high-frequency jitter components that represent high-frequency mechanical vibration and random jitter, and only compensates for the filtered high-frequency jitter components, outputting a stable affine transformation matrix after motion smoothing of the optimal affine transformation matrix.

[0082] It should be noted that the function of the EWMA filter is to separate the high-frequency jitter and low-frequency autonomous flight motion in the composite motion. While eliminating jitter, it retains the intentional heading changes and path tracking motion of the unmanned equipment, thus avoiding the undesirable visual effect of image stickiness in the video output.

[0083] It should be noted that motion smoothing is based on a fixed-length optimal affine transformation matrix sequence ending at the current time. The EWMA filter parameters are dynamically adjusted by the frequency characteristics of the optimal affine transformation matrix sequence to adapt to jitter scenarios of different intensities.

[0084] S5: Perform geometric transformations on the image frame based on the stable affine transformation matrix to generate a stable image frame;

[0085] Specifically, for the current image frame, the stable affine transformation matrix is ​​used as the affine transformation parameter, and the bilinear interpolation algorithm is used to perform a pixel-by-pixel coordinate transformation on the current image frame, projecting the original unstable image frame onto the stable virtual camera coordinate system to generate the corresponding intermediate image frame.

[0086] It should be noted that the geometric transformation process inevitably introduces invalid boundary regions, which are usually manifested as black borders, caused by the reverse translation and rotation of the image plane to compensate for jitter.

[0087] Adaptive cropping is performed on the intermediate image frames after geometric transformation. The boundary between the effective content region and the invalid black border in the image is identified by the edge detection algorithm, and the maximum inscribed rectangle of the effective content region is calculated. The maximum inscribed rectangle region is used as the cropping region. The Lanczos resampling algorithm is used to scale the cropped stable image to the preset output resolution to generate the corresponding stable image frame.

[0088] It should be noted that this step completes the final closed loop of the electronic image stabilization processing chain. It achieves physical compensation for jitter through precise geometric transformation and eliminates visual defects caused by the transformation through adaptive post-processing technology. Finally, it outputs a high-quality, jitter-free, and compliant stable video stream, which meets the hard requirements of agricultural unmanned equipment for subsequent analysis and display of the operation footage.

[0089] The technical solution of this invention is as follows: a dual-channel data acquisition pipeline is constructed and used to acquire image frame data streams and inertial data streams. Based on the inertial data streams, inter-frame basic motion estimation and absolute attitude calculation are performed in real time. The relative three-dimensional rotation transformation and absolute attitude angle matrices are solved and fused to construct a composite affine transformation matrix. Combining the image frame data streams and the composite affine transformation matrix, it is determined in real time whether visual auxiliary data fusion is triggered, and the optimal affine transformation matrix is ​​generated. If not triggered, the IMU dominance duration is recorded. Based on the IMU dominance duration, it is determined whether to trigger the motion prediction correction update of the optimal affine transformation matrix, and motion smoothing processing is performed to output a stable affine transformation matrix. Based on the stable affine transformation matrix, a geometric transformation is performed on the image frame to generate a stable image frame.

[0090] Example 2

[0091] Please see Figure 1 As shown in the embodiment of the present invention, a lightweight electronic image stabilization system based on EIS includes the following modules:

[0092] Acquisition module: Constructs and employs a dual-channel data acquisition pipeline to acquire image frame data streams and inertial data streams;

[0093] IMU solution module: Based on inertial data stream, it performs inter-frame basic motion estimation and absolute attitude solution in real time, solves for relative three-dimensional rotation transformation and absolute attitude angle matrix respectively, and fuses them to construct composite affine transformation matrix;

[0094] Visual assistance module: Combines image frame data stream with composite affine transformation matrix to determine in real time whether visual assistance data fusion is triggered, generates the optimal affine transformation matrix, and records the IMU dominance duration when not triggered;

[0095] Motion Correction and Smoothing Module: Determines whether to trigger motion prediction correction and update of the optimal affine transformation matrix based on the IMU dominance duration, and performs motion smoothing to output a stable affine transformation matrix;

[0096] Image processing module: Performs geometric transformations on image frames based on a stable affine transformation matrix to generate stable image frames.

[0097] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the present invention should still fall within the scope of the present invention.

Claims

1. A lightweight electronic image stabilization method based on EIS, characterized in that: include: A dual-channel data acquisition pipeline was constructed and used to acquire image frame data streams and inertial data streams; Based on inertial data stream, real-time inter-frame basic motion estimation and absolute attitude calculation are performed. The relative three-dimensional rotation transformation and absolute attitude angle matrices are solved separately and then fused to construct a composite affine transformation matrix. By combining the image frame data stream with the composite affine transformation matrix, it is determined in real time whether visual auxiliary data fusion is triggered, the optimal affine transformation matrix is ​​generated, and the duration of IMU dominance is recorded when it is not triggered. Based on the duration of IMU dominance, determine whether to trigger the motion prediction correction update of the optimal affine transformation matrix, and perform motion smoothing to output a stable affine transformation matrix; A geometric transformation is performed on the image frame based on the stable affine transformation matrix to generate a stable image frame.

2. The lightweight electronic image stabilization method based on EIS according to claim 1, characterized in that: The method for obtaining relative three-dimensional rotational changes is as follows: The inertial data stream includes gyroscope data and accelerometer data. Gyroscope data from the inertial data stream within the time interval from the previous image frame to the current image frame is acquired to obtain an inter-frame gyroscope data sequence. Based on the inter-frame gyroscope data sequence, a quaternion differential equation is constructed and numerical integration is performed using the fourth-order Runge-Kutta method. The relative three-dimensional rotational change represented in the form of attitude quaternions is obtained by recursively solving the equation.

3. The lightweight electronic image stabilization method based on EIS according to claim 2, characterized in that: The absolute attitude angle matrix is ​​obtained as follows: Based on the gyroscope and accelerometer data included in the inertial data stream, an absolute attitude calculation is performed using a complementary filter fusion algorithm. Low-frequency attitude signals of pitch and roll angles are obtained by calculating the pitch and roll angles using the accelerometer data. Based on the continuous gyroscope data, the high-frequency attitude signals are obtained by integrating quaternion differential equations and converting them into Euler angles. The low-frequency attitude signals and high-frequency attitude signals are then weighted and fused using a complementary filter to output an absolute attitude angle matrix including pitch, yaw, and roll angles.

4. A lightweight electronic image stabilization method based on EIS according to claim 3, characterized in that: The composite affine transformation matrix is ​​constructed as follows: The relative three-dimensional rotational transformation is decomposed and projected onto the two-dimensional image plane to obtain the basic affine transformation used to compensate for inter-frame jitter. Based on the absolute attitude angle matrix, a rotational transformation is generated to actively correct image tilt and maintain the horizon level. The basic affine transformation and the rotational transformation are combined by matrix multiplication to obtain the composite affine transformation matrix.

5. A lightweight electronic image stabilization method based on EIS according to claim 1, characterized in that: The optimal affine transformation matrix is ​​generated as follows: The texture richness of the current image frame is evaluated. If the texture richness is high, fast optical flow calculation is performed on the current image frame and the previous stable image frame to fit the image affine transformation matrix. The stable image frame represents the image frame output after the corresponding image frame has undergone the entire electronic image stabilization process. The Euclidean distance between the composite affine transformation matrix and the image affine transformation matrix in the vector space is calculated. If the Euclidean distance is less than the preset fault tolerance threshold, the IMU dominance duration is set to zero, triggering visual auxiliary data fusion based on Kalman filtering. The composite affine transformation matrix and the image affine transformation matrix are fused to generate the optimal affine transformation matrix. If the Euclidean distance is greater than or equal to the fault tolerance threshold, or if the current texture richness is low, the IMU-dominated timing starts, and the composite affine transformation matrix is ​​used as the optimal affine transformation matrix output.

6. A lightweight electronic image stabilization method based on EIS according to claim 5, characterized in that: The method for evaluating texture richness is as follows: The Laplacian operator is used to perform convolution operation on the current image frame, and the Laplacian variance of the output image is calculated as a quantitative indicator of image texture richness. If the Laplacian variance is lower than the preset variance threshold, the current texture richness is judged to be low; otherwise, the current texture richness is judged to be high.

7. A lightweight electronic image stabilization method based on EIS according to claim 1, characterized in that: The method for updating motion prediction corrections is as follows: Maintain a fixed-length first-in-first-out queue, store the optimal affine transformation matrix sequence, read the three-dimensional flight velocity vector of the unmanned equipment in real time from the flight control system for the corresponding time period, organize it to obtain the flight velocity sequence, obtain the absolute attitude angle matrix of the unmanned equipment for the corresponding time period, and organize it to obtain the body attitude sequence. Time series regression analysis is performed based on the optimal affine transformation matrix sequence to predict the predicted increment changes of each component of the optimal affine transformation matrix. Combined with the flight velocity sequence and the aircraft attitude sequence, the actual increment changes are estimated through kinematic prediction. Deviation analysis is performed between the predicted increment changes and the actual increment changes to obtain the baseline drift trend and perform geometric reversal to obtain the inverse transformation. The inverse transformation is then geometrically synthesized and corrected with the optimal affine transformation matrix of the current image frame to complete the update.

8. A lightweight electronic image stabilization method based on EIS according to claim 7, characterized in that: The stable affine transformation matrix is ​​obtained as follows: An exponentially weighted moving average filter is used as the motion filtering algorithm to perform motion smoothing on the optimal affine transformation matrix or the optimal affine transformation matrix updated by motion prediction. This process retains the low-frequency components that represent the intentional flight path of the unmanned equipment, attenuates the high-frequency jitter components that represent high-frequency mechanical vibration and random jitter, and outputs a stable affine transformation matrix that has undergone motion smoothing.

9. A lightweight electronic image stabilization method based on EIS according to claim 1, characterized in that: The method for generating stable image frames is as follows: Using the stable affine transformation matrix as the affine transformation parameter, the current image frame is transformed pixel by pixel using the bilinear interpolation algorithm to generate an intermediate image frame. The intermediate image frame after geometric transformation is adaptively cropped to identify the effective content region and calculate the maximum inscribed rectangle. The Lanczos resampling algorithm is then used to scale the cropped stable image to the preset output resolution to generate a stable image frame.

10. A lightweight electronic image stabilization system based on EIS, characterized in that: Includes the following modules: Acquisition module: Constructs and employs a dual-channel data acquisition pipeline to acquire image frame data streams and inertial data streams; IMU solution module: Based on inertial data stream, it performs inter-frame basic motion estimation and absolute attitude solution in real time, solves for relative three-dimensional rotation transformation and absolute attitude angle matrix respectively, and fuses them to construct composite affine transformation matrix; Visual assistance module: Combines image frame data stream with composite affine transformation matrix to determine in real time whether visual assistance data fusion is triggered, generates the optimal affine transformation matrix, and records the IMU dominance duration when not triggered; Motion Correction and Smoothing Module: Determines whether to trigger motion prediction correction and update of the optimal affine transformation matrix based on the IMU dominance duration, and performs motion smoothing to output a stable affine transformation matrix; Image processing module: Performs geometric transformations on image frames based on a stable affine transformation matrix to generate stable image frames.

Images