Electronic anti-shake method and system based on IMU (Inertial Measurement Unit) and field angle

Abstract

The invention provides an electronic anti-shake method and system based on an IMU (Inertial Measurement Unit) and a field angle, and the method comprises the following steps: collecting IMU data in a process that imaging equipment obtains a real-time video stream; according to the IMU data between every two adjacent frames of images, the angle variation between the two adjacent frames of images is obtained; calculating a pixel offset between every two adjacent frames of images according to the angle variation, the field angle and the image resolution; accumulating pixel offsets between every two adjacent frames of images to determine an original motion track and a smooth motion track of the imaging equipment; and determining a jitter compensation amount according to the original motion trail and the smooth motion trail, and performing jitter compensation on the image according to the jitter compensation amount. According to the invention, the calculation of the pixel offset and the jitter compensation amount can be rapidly completed only based on the IMU data and the field angles under different focal lengths, the overall size and weight of the device can be greatly reduced, and the method is suitable for jitter compensation of different zoom positions.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an electronic image stabilization method and system based on IMU and field of view. Background Technology

[0002] Currently, during the use of infrared imaging equipment, external forces can cause the equipment to shake, which in turn affects the imaging effect. Therefore, corresponding anti-shake and image stabilization measures are needed to ensure image quality.

[0003] Existing image stabilization technologies mainly include optical image stabilization, mechanical image stabilization, and electronic image stabilization, among which:

[0004] Optical image stabilization uses sensors such as gyroscopes inside the lens of an imaging device to detect the shaking of the device, and then compensates for the shaking by moving one or more sets of lenses in the lens. However, optical image stabilization is mainly suitable for application environments with small shaking amplitude, and requires the configuration of high-precision sensors, which leads to high technical difficulty, complex mechanical structure and high hardware cost.

[0005] Mechanical image stabilization is mainly based on mechanical structure. It records the shaking of the device through sensors such as gyroscopes, and then adjusts the servo motor system in reverse according to the shaking. The stabilization effect is achieved by mechanical transmission of the gimbal. However, mechanical image stabilization has low image stabilization accuracy, and external gimbal devices are generally too large, which does not meet the design requirements of miniaturization.

[0006] Electronic image stabilization (EIS) is achieved through software algorithms. It analyzes continuously captured image frames, calculates the device's shake trajectory, and then applies image stabilization. For example, motion estimation based on image grayscale projection or feature matching, followed by compensation, is computationally intensive and time-consuming, and unsuitable for scenarios with fast-moving devices. Alternatively, it can quickly estimate inter-frame motion based on homography matrices, but calculating the homography matrix requires knowing the camera's intrinsic parameters and the relative positions of the camera and IMU (Inertial Measurement Unit). Furthermore, during zooming of the imaging device's lens, the camera's intrinsic parameters change at different focal lengths, necessitating multiple calibrations and complicating the process. Summary of the Invention

[0007] The purpose of this invention is to provide an electronic image stabilization method and system based on IMU and field of view. It can quickly calculate the pixel offset and shake compensation based only on IMU data and the field of view at different focal lengths, which can greatly reduce the overall size and weight of the device and is applicable to shake compensation at different zoom positions.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] On the one hand, an electronic image stabilization method based on IMU and field of view is provided, which includes the following steps:

[0010] IMU data is acquired during the process of the imaging device acquiring a real-time video stream;

[0011] The angle change Δθ between two adjacent frames is obtained based on the IMU data between each pair of adjacent frames.

[0012] The pixel offset d between each pair of adjacent frames is calculated based on the angle change Δθ between each pair of adjacent frames, the field of view (FOV) of the imaging device when acquiring the current video stream, and the image resolution.

[0013] The pixel offset d between each two adjacent frames is accumulated to determine the original motion trajectory Traj[n] and the smooth motion trajectory Traj_s[n] of the imaging device;

[0014] The jitter compensation amount Δstab[n] is determined based on the original motion trajectory Traj[n] and the smooth motion trajectory Traj_s[n], and jitter compensation is performed on the image based on the jitter compensation amount Δstab[n].

[0015] Preferably, IMU data is acquired simultaneously whenever a valid field signal is obtained.

[0016] Preferably, the IMU data acquisition frequency f IMU , where f I K represents the acquisition frequency of the original image data, and K is a positive integer with a value range of [10, 20].

[0017] Preferably, the angle change Δθ between any two adjacent frames is obtained through the following steps:

[0018] Determine the output time of the image corresponding to each piece of raw image data;

[0019] The IMU data between two adjacent frames is integrated to obtain the angular change Δθ between the two adjacent frames, and Δθ=∫ωdt, where ω is the angular velocity and t is the time between the two adjacent frames.

[0020] Preferably, the IMU data between two adjacent frames undergoes zero-bias compensation and filtering.

[0021] Preferably, the field of view (FOV) of the imaging device when acquiring the current video stream is obtained through the following steps:

[0022] A mapping relationship between the focal length and field of view (FOV) of the imaging device is pre-constructed, and the mapping relationship includes different focal lengths and the field of view corresponding to each focal length;

[0023] The position of the zoom motor when the imaging device acquires the current video stream is obtained in real time, and the current focal length is determined based on the current position of the zoom motor.

[0024] The field of view (FOV) corresponding to the current focal length is determined based on the mapping relationship between focal length and field of view, so as to serve as the FOV when the imaging device acquires the current video stream.

[0025] Preferably, the pixel offset between every two adjacent frames .

[0026] Preferably, the original motion trajectory n is the total number of images in the current video stream, and i represents the i-th frame image;

[0027] And, if n < m, then the smooth motion trajectory If n≥m, then Where m is the number of smooth frames;

[0028] Preferably, the jitter compensation amount .

[0029] On the other hand, an electronic image stabilization system is also provided, which includes:

[0030] An IMU is used to simultaneously acquire IMU data while obtaining a valid field signal;

[0031] The field of view (FOV) acquisition unit is used to determine the FOV of the imaging device when acquiring the current video stream;

[0032] An angle calculation unit is used to obtain the angle change Δθ between two adjacent frames based on the IMU data between each two adjacent frames.

[0033] The offset calculation unit is used to calculate the pixel offset d between each pair of adjacent frames based on the angular change Δθ between each pair of adjacent frames, the field of view (FOV), and the image resolution.

[0034] The compensation calculation unit is used to accumulate the pixel offset d between each two adjacent frames of images to determine the original motion trajectory Traj[n] of the imaging device, and to perform smoothing filtering on the original motion trajectory Traj[n] to obtain the smooth motion trajectory Traj_s[n] of the imaging device, and to determine the jitter compensation amount Δstab[n] based on the original motion trajectory Traj[n] and the smooth motion trajectory Traj_s[n].

[0035] And a jitter compensation unit, which is used to perform jitter compensation on the image according to the jitter compensation amount Δstab[n] and output the compensated image.

[0036] In summary, the present invention has the following advantages compared with the prior art:

[0037] This invention can quickly calculate pixel offset and shake compensation based solely on IMU data and field of view at different focal lengths. It eliminates the need for motion estimation through complex image feature matching and the need to set up complex mechanical image stabilization structures, which can greatly reduce the overall size and weight of the device. Furthermore, it eliminates the need for multiple intrinsic parameter calibrations of the imaging device, significantly reducing the computational load, and is applicable to shake compensation at different zoom positions. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating the steps of the electronic image stabilization method based on IMU and field of view in this invention.

[0039] Figure 2 Images before and after processing by the electronic image stabilization method of this invention;

[0040] Figure 3 This is a schematic diagram of the electronic image stabilization method in this invention. Detailed Implementation

[0041] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0042] Example 1

[0043] like Figure 1 As shown, this embodiment provides an electronic image stabilization method based on IMU and field of view, which includes the following steps:

[0044] S1. During the process of the imaging device acquiring real-time video stream, the field valid signal is acquired in real time, and IMU data is acquired synchronously whenever a field valid signal is acquired, and the acquired IMU data is cached.

[0045] In this embodiment, the imaging device includes an infrared imaging device, and the acquisition of the effective field signal can be completed based on the image acquisition chip / timing control module of the imaging device, and the IMU data is acquired based on the LSM6D type IMU.

[0046] Furthermore, acquiring a real-time video stream includes the following steps:

[0047] Raw image data is acquired by an image detector (such as an infrared detector). The raw image data can be infrared raw data (such as infrared x16 data) or visible light raw data (such as NumPy arrays), for example, y8 data or yuv data.

[0048] The original image data is preprocessed, including two-point correction, bad pixel removal, noise reduction, dimming, image sharpening, and image enhancement.

[0049] The image detector converts pre-processed raw image data into a visual image, ultimately outputting an infrared or visible light image.

[0050] S2. Construct a frame dataset, which contains frame data corresponding to each image in the video stream. The frame data includes original image data (such as y16 data, y8 data, yuv data, etc. of infrared images), images generated based on the original image data, and IMU data collected at the time the original image data is acquired. Thus, the original image data and IMU data can be superimposed to synchronize the original image data and IMU data.

[0051] In this embodiment, the IMU data acquisition frequency f IMU , where f I The original image data acquisition frequency is [30Hz, 50Hz]; K is a positive integer with a value range of [10, 20].

[0052] Therefore, in this embodiment, the IMU data acquisition frequency f IMU f is the frequency of raw image data acquisition. I The number of IMU data is an integer multiple of the number of original image data, so that at each time the original image data is acquired, a frame of IMU data at that time is collected, ensuring that each original image data has a corresponding frame of IMU data.

[0053] S3. Time-align the images and IMU data within each frame of the frame dataset, specifically including the following steps:

[0054] At least one frame of original image data is edited manually, and the time t0 when the editing is completed is recorded. The editing includes data modification and replacement. For example, one frame of infrared x16 data contains a number of elements equal to the image width. The image height is short data, and the value corresponding to each pixel is not fixed. Therefore, the short data can be modified to make it significantly different from the original data.

[0055] The image detector outputs an updated image based on the edited original image data, and records the output time t1 of the updated image;

[0056] The delay Δt is obtained based on Δt = t1 - t0;

[0057] In the frame dataset, the output time of the original image data in each frame is added with a delay Δt to obtain the output time of the image corresponding to each original image data.

[0058] This allows us to determine the output time of the image and the acquisition time of the IMU data within each frame of the frame dataset, thus achieving time alignment between the image and the IMU data.

[0059] Since the original image data needs to be processed and transformed before the corresponding image can be generated for display, there are delays such as algorithm time and data transmission time in this process, which causes the image output time to be inconsistent with the IMU data acquisition time. Therefore, it is necessary to determine the output time of each image so that it is aligned with the IMU data acquisition time.

[0060] Furthermore, the original image data of each frame in the video stream may be very similar, resulting in little difference between the generated images. Therefore, it is impossible to quickly locate a specific frame in a series of images and determine its output time. In this embodiment, the original image data is edited so that the updated image output based on the edited original image data is significantly different from the original image (i.e., the image generated based on the unedited original image data). Thus, the updated image can be quickly located from the video stream images, and the delay Δt can be calculated based on the output time t1 of the updated image.

[0061] Finally, the output time of the original image data in each frame is delayed by Δt to obtain the output time of the image corresponding to each original image data, thereby completing the time alignment of the image and IMU data.

[0062] S4. Determine the field of view (FOV) of the imaging device when acquiring the current video stream, which specifically includes the following steps:

[0063] S41. A mapping relationship between the focal length and field of view (FOV) of the imaging device is pre-constructed, and the mapping relationship includes different focal lengths and the field of view corresponding to each focal length.

[0064] For example, in this embodiment, the imaging device can be aligned with the same target, and then the focal length of the imaging device can be adjusted. The field of view (FOV) corresponding to each focal length can be determined, thereby obtaining the mapping relationship between focal length and field of view (FOV). This mapping relationship can be displayed in the form of tables, curves, etc.

[0065] S42. Real-time acquisition of the zoom motor position when the imaging device acquires the current video stream, and determination of the current focal length based on the current zoom motor position;

[0066] The zoom motor drives the movement of the lens group to achieve lens zoom of the imaging device. Therefore, there is a corresponding relationship between the position of the zoom motor and the focal length. So, after obtaining the current position of the zoom motor, the current focal length can be determined by the correspondence between the two. In this embodiment, the position of the zoom motor can be obtained by components such as the Hall sensor mounted on the imaging device.

[0067] S43. Determine the field of view corresponding to the current focal length based on the mapping relationship between focal length and field of view, which is the field of view when the imaging device acquires the current video stream; for example, the field of view (FOV) corresponding to the current focal length can be determined by querying a table.

[0068] S5. Obtain the angle change Δθ between two adjacent frames based on the IMU data between each two adjacent frames, and calculate the pixel offset d between each two adjacent frames based on the angle change Δθ between each two adjacent frames, the field of view (FOV), and the image resolution.

[0069] The angle change Δθ between two adjacent frames is obtained through the following steps:

[0070] S51. Use tools such as IMU_UTILS to calibrate the IMU to obtain the IMU's bias and noise, thereby determining the source of error in the IMU data;

[0071] Additionally, zero-bias compensation and filtering are performed on the IMU data between two adjacent frames of images acquired in real time to remove noise from the IMU.

[0072] Since the output time of each image has been obtained in the previous step, the IMU data between two adjacent frames can be determined.

[0073] S52. Perform integral calculation on the IMU data between two adjacent frames that have undergone zero bias compensation and filtering to obtain the angle change Δθ between the two adjacent frames.

[0074] In this embodiment, Δθ=∫ωdt, where ω is the angular velocity, which can be obtained from the angular velocity data of the three-axis gyroscope in the IMU data; t is the time between two adjacent frames.

[0075] Furthermore, ;

[0076] By repeating the above steps, the angle change Δθ and pixel offset d between each two adjacent frames can be obtained;

[0077] S6. Accumulate the pixel offset d between each two adjacent frames to determine the original motion trajectory Traj[n] of the imaging device, and perform smoothing filtering on the original motion trajectory Traj[n] to obtain the smooth motion trajectory Traj_s[n] of the imaging device. The difference between the original motion trajectory Traj[n] and the smooth motion trajectory Traj_s[n] is used as the jitter compensation amount Δstab[n] of the imaging device.

[0078] Specifically, the original motion trajectory n is the total number of images in the current video stream, and i represents the i-th frame image;

[0079] If n < m, then the smooth motion trajectory If n≥m, then Where m is the number of smooth frames;

[0080] Furthermore, the jitter compensation amount ;

[0081] And, S7, perform jitter compensation on the image according to the jitter compensation amount Δstab[n], and output the compensated image, thereby completing the jitter compensation for the real-time video stream.

[0082] like Figure 2 As shown, after processing by the electronic image stabilization method in this embodiment, the three consecutive frames (a)-(c) in the output video stream of the imaging device are brighter and have clearer details than the images (e)-(g) output without processing by the electronic image stabilization method in this embodiment. This shows that the electronic image stabilization method in this embodiment can achieve a good image stabilization effect.

[0083] Therefore, compared with the prior art, the electronic image stabilization method in this embodiment does not require a complex mechanical image stabilization structure, which can greatly reduce the overall size and weight of the device. It only needs to calculate the pixel offset and shake compensation amount based on IMU data and the field of view at different focal lengths. It does not require motion estimation through complex image feature matching and completes image shake compensation based on the shake compensation amount. This process does not require multiple intrinsic parameter calibrations of the imaging device, which can greatly reduce the amount of computation and is applicable to shake compensation at different zoom positions.

[0084] Example 2:

[0085] The only difference between this embodiment and embodiment 1 is that in step S7, before outputting the compensated image, the black border area of ​​the compensated image is filled by edge pixels, and / or the compensated image is cropped and scaled before being output.

[0086] Example 3:

[0087] This embodiment provides an electronic image stabilization system that can implement the electronic image stabilization method described in Embodiment 1 or 2, such as... Figure 3 As shown, the electronic image stabilization system includes:

[0088] IMU 1 is used to simultaneously acquire IMU data while obtaining a valid field signal, and the IMU data acquisition frequency f IMU

[0089] The delay calculation unit 2 is used to determine the output time of each image in the video stream in order to complete the time alignment of the image and IMU data. Its process is the same as step S3.

[0090] The field of view acquisition unit 3 is used to determine the field of view (FOV) when the imaging device acquires the current video stream, and its process is the same as step S4.

[0091] Angle calculation unit 4 is used to obtain the angle change Δθ between two adjacent frames based on the IMU data between each two adjacent frames. The process is the same as step S5.

[0092] Offset calculation unit 5 is used to calculate the pixel offset d between each two adjacent frames based on the angle change Δθ between each two adjacent frames, the field of view (FOV), and the image resolution. The process is the same as step S5.

[0093] The compensation calculation unit 6 is used to accumulate the pixel offset d between each two adjacent frames of images to determine the original motion trajectory Traj[n] of the imaging device, and to perform smoothing filtering on the original motion trajectory Traj[n] to obtain the smooth motion trajectory Traj_s[n] of the imaging device. The difference between the original motion trajectory Traj[n] and the smooth motion trajectory Traj_s[n] is used as the jitter compensation amount Δstab[n] of the imaging device. The process is the same as step S6.

[0094] The jitter compensation unit 7 is used to perform jitter compensation on the image according to the jitter compensation amount Δstab[n] and output the compensated image.

[0095] In summary, this invention can quickly calculate pixel offset and shake compensation based solely on IMU data and field of view at different focal lengths, without requiring complex image feature matching for motion estimation or complex mechanical image stabilization structures. This greatly reduces the overall size and weight of the device, eliminates the need for multiple intrinsic parameter calibrations of the imaging device, significantly reduces computational load, and is applicable to shake compensation at different zoom positions.

[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An electronic image stabilization method based on IMU and field of view, characterized in that, Includes the following steps: IMU data is acquired during the process of the imaging device acquiring a real-time video stream; The angle change Δθ between two adjacent frames is obtained based on the IMU data between each pair of adjacent frames. The pixel offset d between each pair of adjacent frames is calculated based on the angle change Δθ between each pair of adjacent frames, the field of view (FOV) of the imaging device when acquiring the current video stream, and the image resolution. The pixel offset d between each two adjacent frames is accumulated to determine the original motion trajectory Traj[n] and the smooth motion trajectory Traj_s[n] of the imaging device; The jitter compensation amount Δstab[n] is determined based on the original motion trajectory Traj[n] and the smooth motion trajectory Traj_s[n], and jitter compensation is performed on the image based on the jitter compensation amount Δstab[n].

2. The electronic image stabilization method as described in claim 1, characterized in that, Whenever a valid field signal is obtained, IMU data is acquired simultaneously.

3. The electronic image stabilization method as described in claim 1, characterized in that, IMU data acquisition frequency f IMU , where f I K represents the acquisition frequency of the original image data, and K is a positive integer with a value range of [10, 20].

4. The electronic image stabilization method as described in claim 1, characterized in that, The angle change Δθ between each two adjacent frames is obtained through the following steps: Determine the output time of the image corresponding to each piece of raw image data; The IMU data between two adjacent frames is integrated to obtain the angular change Δθ between the two adjacent frames, and Δθ=∫ωdt, where ω is the angular velocity and t is the time between the two adjacent frames.

5. The electronic image stabilization method as described in claim 4, characterized in that, The IMU data between two adjacent frames undergoes zero-bias compensation and filtering.

6. The electronic image stabilization method as described in claim 1, characterized in that, The imaging device acquires the field of view (FOV) of the current video stream through the following steps: A mapping relationship between the focal length and field of view (FOV) of the imaging device is pre-constructed, and the mapping relationship includes different focal lengths and the field of view corresponding to each focal length; The position of the zoom motor when the imaging device acquires the current video stream is obtained in real time, and the current focal length is determined based on the current position of the zoom motor. The field of view (FOV) corresponding to the current focal length is determined based on the mapping relationship between focal length and field of view, so as to serve as the FOV when the imaging device acquires the current video stream.

7. The electronic image stabilization method as described in claim 1, characterized in that, Pixel offset between two adjacent frames .

8. The electronic image stabilization method as described in claim 1, characterized in that, The original motion trajectory n is the total number of images in the current video stream, and i represents the i-th frame image; And, if n < m, then the smooth motion trajectory If n≥m, then , where m is the number of smooth frames.

9. The electronic image stabilization method as described in claim 8, characterized in that, The jitter compensation amount .

10. An electronic image stabilization system, characterized in that, include: An IMU is used to simultaneously acquire IMU data while obtaining a valid field signal; The field of view (FOV) acquisition unit is used to determine the FOV of the imaging device when acquiring the current video stream; An angle calculation unit is used to obtain the angle change Δθ between two adjacent frames based on the IMU data between each two adjacent frames. The offset calculation unit is used to calculate the pixel offset d between each pair of adjacent frames based on the angular change Δθ between each pair of adjacent frames, the field of view (FOV), and the image resolution. The compensation calculation unit is used to accumulate the pixel offset d between each two adjacent frames of images to determine the original motion trajectory Traj[n] of the imaging device, and to perform smoothing filtering on the original motion trajectory Traj[n] to obtain the smooth motion trajectory Traj_s[n] of the imaging device, and to determine the jitter compensation amount Δstab[n] based on the original motion trajectory Traj[n] and the smooth motion trajectory Traj_s[n]. And a jitter compensation unit, which is used to perform jitter compensation on the image according to the jitter compensation amount Δstab[n] and output the compensated image.

Images