Electronic image stabilization system based on gyro residual error

Abstract

The invention relates to the technical field of electronic image stabilization, in particular to an electronic image stabilization system based on gyro residual errors. Comprising a gyroscope data acquisition module, an image acquisition matching module, a global motion estimation module, a gyroscope residual calculation module, a residual compensation module, an image stability compensation module and an image sequence output module, the gyro data acquisition module is used for acquiring angular velocity data of a gyro sensor; the image acquisition matching module is used for matching position changes of the feature points and estimating local motion parameters between the images; the global motion estimation module calculates global motion parameters of the image sequence; a gyro residual calculation module calculates a residual between a global motion estimation result and a local motion parameter; the residual compensation module compensates and corrects global motion estimation; the image stabilization compensation module performs stabilization compensation on the image sequence; through the mode, the precision and the reliability of electronic image stabilization are effectively improved, and the method is particularly suitable for complex motion scenes and application occasions with high image stabilization precision requirements.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic image stabilization, and particularly to an electronic image stabilization system based on gyro residual. Background Art

[0002] In many fields such as video surveillance, vehicle-mounted cameras, aerospace imaging, etc., due to the movement of the shooting device carrier, such as the bumps during vehicle driving, vibrations during aircraft flight and other complex scenarios, the captured image sequence will appear jitter phenomenon, seriously affecting the quality of the image and subsequent analysis and processing. The electronic image stabilization technology aims to perform stable processing on the jittery image sequence through image processing algorithms to obtain clear and stable images. Traditional electronic image stabilization methods mainly estimate the motion parameters between image sequences based on image feature matching, and then compensate the images. However, in complex scenarios, such as low texture, large illumination changes, serious occlusion, etc., the accuracy of feature extraction and matching will be greatly affected, resulting in poor image stabilization effect. Moreover, relying solely on image information for image stabilization has limited ability to compensate for fast and large-amplitude jitters.

[0003] Gyro sensors can directly measure the angular velocity information of the carrier. Applying it to the electronic image stabilization system can, to a certain extent, overcome the deficiencies of the method based on image feature matching. However, due to problems such as measurement errors and drifts of the gyro, directly using the gyro measurement data for image stabilization cannot achieve ideal results.

[0004] In summary, it is very necessary to propose an electronic image stabilization system based on gyro residual to improve the image stabilization effect in complex scenarios and avoid errors caused by solely relying on gyro information. Summary of the Invention

[0005] The purpose of the present invention is to provide an electronic image stabilization system based on gyro residual, which solves the problems of unsatisfactory image stabilization effect of the existing electronic image stabilization technology in complex scenarios and errors caused by solely relying on gyro information.

[0006] To achieve the above purpose, an electronic image stabilization system based on gyro residual adopted by the present invention includes a gyro data acquisition module, an image acquisition and matching module, a global motion estimation module, a gyro residual calculation module, a residual compensation module, an image stabilization compensation module, and an image sequence output module. The global motion estimation module is respectively connected to the gyro data acquisition module, the image acquisition and matching module, and the gyro residual calculation module. The gyro residual calculation module is respectively connected to the image acquisition and matching module and the residual compensation module. The image stabilization compensation module is respectively connected to the residual compensation module, the image acquisition and matching module, and the image sequence output module;

[0007] The gyro data acquisition module is used to collect the angular velocity data of the gyro sensor and perform data preprocessing;

[0008] The image acquisition and matching module is used to extract features from the original jittery image sequence, match the position changes of feature points, and estimate the local motion parameters between images;

[0009] The global motion estimation module is used to calculate the global motion parameters of the image sequence based on the preprocessed gyro data;

[0010] The gyro residual calculation module is used to compare the global motion estimation result based on the gyro with the local motion parameters obtained by image feature matching, and calculate the residual between the two;

[0011] The residual compensation module is used to compensate and correct the global motion estimation based on the gyro according to the calculated gyro residual;

[0012] The image stabilization compensation module is used to perform stabilization compensation on the image sequence according to the global motion parameters after residual compensation;

[0013] The image sequence output module is used to output the stabilized image sequence.

[0014] Among them, the image acquisition and matching module includes an image acquisition unit and an image feature extraction and matching unit. The image acquisition unit is respectively connected to the image feature extraction and matching unit and the global motion estimation module. The image feature extraction and matching unit is respectively connected to the gyro residual calculation module and the image stabilization compensation module.

[0015] Among them, the image acquisition unit is used to collect the original jittery image sequence and transmit the image data;

[0016] The image feature extraction and matching unit is used to extract features from the collected image sequence, match the feature points in adjacent image frames, and estimate the local motion parameters between images by the position changes of the matched feature points.

[0017] Among them, the gyro data acquisition module includes a gyro data collection unit and a gyro data preprocessing unit. The gyro data preprocessing unit is respectively connected to the gyro data collection unit and the global motion estimation module.

[0018] Among them, the gyro data collection unit is used to collect the angular velocity data of the gyro sensor;

[0019] The gyro data preprocessing unit is used to perform preprocessing on the collected gyro angular velocity data.

[0020] An electronic image stabilization system based on gyro residual of the present invention, wherein the gyro data acquisition module is used to collect the angular velocity data of the gyro sensor and perform data preprocessing; the image acquisition and matching module is used to extract features from the original jittery image sequence, match the position changes of feature points, and estimate the local motion parameters between images; the global motion estimation module is used to calculate the global motion parameters of the image sequence according to the preprocessed gyro data; the gyro residual calculation module is used to compare the global motion estimation result based on the gyro with the local motion parameters obtained by image feature matching, and calculate the residual between the two; the residual compensation module is used to compensate and correct the global motion estimation based on the gyro according to the calculated gyro residual; the image stabilization compensation module is used to perform stabilization compensation on the image sequence according to the global motion parameters after residual compensation; the image sequence output module is used to output the stabilized image sequence; which improves the image stabilization effect in complex scenarios and avoids errors caused by solely relying on gyro information, effectively improving the accuracy and reliability of electronic image stabilization, and is particularly suitable for complex motion scenarios and application occasions with high requirements for image stabilization accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0022] Figure 1 FIG. is a schematic diagram of the principle of the electronic image stabilization system based on gyro residual of the present invention.

[0023] 100 - Gyro data acquisition unit, 200 - Image acquisition unit, 300 - Gyro data preprocessing unit, 400 - Global motion estimation module, 500 - Image feature extraction and matching unit, 600 - Gyro residual calculation module, 700 - Residual compensation module, 800 - Image stabilization compensation module, 900 - Image sequence output module. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] Please refer to Figure 1 , wherein Figure 1 is a schematic diagram of the principle of the electronic image stabilization system based on gyro residual.

[0025] The present invention provides an electronic image stabilization system based on gyro residual, which includes a gyro data acquisition module, an image acquisition and matching module, a global motion estimation module 400, a gyro residual calculation module 600, a residual compensation module 700, an image stabilization compensation module 800, and an image sequence output module 900. The global motion estimation module 400 is respectively connected to the gyro data acquisition module, the image acquisition and matching module, and the gyro residual calculation module 600. The gyro residual calculation module 600 is respectively connected to the image acquisition and matching module and the residual compensation module 700. The image stabilization compensation module 800 is respectively connected to the residual compensation module 700, the image acquisition and matching module, and the image sequence output module 900;

[0026] The gyro data acquisition module is configured to collect the angular velocity data of the gyro sensor and perform data preprocessing;

[0027] The image acquisition and matching module is configured to extract features from the original jittery image sequence, match the position changes of the feature points, and estimate the local motion parameters between the images;

[0028] The global motion estimation module 400 is configured to calculate the global motion parameters of the image sequence according to the preprocessed gyro data;

[0029] The gyro residual calculation module 600 is configured to compare the global motion estimation result based on the gyro with the local motion parameters obtained by image feature matching, and calculate the residual between the two;

[0030] The residual compensation module 700 is configured to compensate and correct the global motion estimation based on the gyro according to the calculated gyro residual;

[0031] The image stabilization compensation module 800 is configured to perform stabilization compensation on the image sequence according to the global motion parameters after residual compensation;

[0032] The image sequence output module 900 is configured to output the stabilized image sequence.

[0033] In this embodiment, the gyro data acquisition module collects the angular velocity data of the gyro sensor and performs data preprocessing. The image acquisition and matching module extracts features from the original jittery image sequence, matches the position changes of feature points, and estimates the local motion parameters between images. The global motion estimation module 400 is configured to calculate the global motion parameters of the image sequence according to the preprocessed gyro data; by integrating the gyro angular velocity data over time, global motion information such as the rotation angle and translation amount of the image sequence at each moment relative to the initial moment is obtained, so as to estimate the image jitter caused by the carrier motion. The gyro residual calculation module 600 compares the global motion estimation result based on the gyro with the local motion parameters obtained by image feature matching, and calculates the residual between the two; this residual reflects the difference between the global motion estimated by the gyro and the actual local motion of the image, and its main sources include the measurement error of the gyro, the error of image feature matching, and the difference caused by non-rigid motion factors such as scene changes. The residual compensation module 700 compensates and corrects the global motion estimation based on the gyro according to the calculated gyro residual; the weighted average method is used to fuse the gyro residual information into the global motion estimation to obtain more accurate global motion parameters of the image sequence. The image stabilization compensation module 800 performs stabilization compensation on the image sequence according to the global motion parameters after residual compensation; by performing reverse rotation matrix transformation and translation transformation on the image, the jittery image is restored to a relatively stable state. The image sequence output module 900 is configured to output the stabilized image sequence. By the above method, the image stabilization effect in a complex scene is improved and the error caused by simply relying on gyro information is avoided, effectively improving the accuracy and reliability of electronic image stabilization, especially suitable for complex motion scenes and application occasions with high requirements for image stabilization accuracy.

[0034] Further, the image acquisition and matching module includes an image acquisition unit 200 and an image feature extraction and matching unit 500. The image acquisition unit 200 is respectively connected to the image feature extraction and matching unit 500 and the global motion estimation module 400, and the image feature extraction and matching unit 500 is respectively connected to the gyro residual calculation module 600 and the image stabilization compensation module 800.

[0035] Further, the image acquisition unit 200 is configured to collect the original jittery image sequence and transmit the image data.

[0036] The image feature extraction and matching unit 500 is configured to extract features from the collected image sequence, match the feature points in adjacent image frames, and estimate the local motion parameters between images by the position changes of the matched feature points.

[0037] In this embodiment, the image acquisition unit 200 is configured to acquire an original jitter image sequence and transmit the image data. The image feature extraction and matching unit 500 divides the image into 16 regions of 4x4, and respectively adopts the SURF feature extraction algorithm to achieve fast and effective extraction of significant image features, optimize the extracted feature points, remove noise points and redundant points, and then match the feature points in adjacent image frames. The local motion parameters between images are estimated by the position changes of the matched feature points.

[0038] Further, the gyro data acquisition module includes a gyro data acquisition unit 100 and a gyro data preprocessing unit 300. The gyro data preprocessing unit 300 is respectively connected to the gyro data acquisition unit 100 and the global motion estimation module 400.

[0039] Further, the gyro data acquisition unit 100 is configured to acquire the angular velocity data of the gyro sensor;

[0040] The gyro data preprocessing unit 300 is configured to preprocess the acquired gyro angular velocity data.

[0041] In this embodiment, the gyro data acquisition unit 100 acquires the angular velocity data of the gyro sensor. The gyro data acquisition unit 100 includes a gyro sensor and peripheral circuits, which are kept parallel to the carrier imaging sensor, and acquires the angular velocity values of the carrier in the X, Y, and Z axes at a sampling frequency of 1000 hz. The gyro data preprocessing unit 300 preprocesses the acquired gyro angular velocity data, including removing noise interference, and using the Kalman filter algorithm to smooth the gyro data to reduce the influence of measurement noise and drift on the data; at the same time, according to the initial calibration parameters of the gyro, the gyro data is corrected to convert it into relatively accurate carrier angular velocity information.

[0042] The above-disclosed are only one or more preferred embodiments of the present invention. Of course, the scope of the rights of the present invention cannot be limited thereby. Those of ordinary skill in the art can understand all or part of the processes of implementing the above embodiments, and the equivalent changes made according to the claims of the present invention still fall within the scope covered by the present invention.

Claims

1. An electronic image stabilization system based on gyro residual, characterized in that: It includes a gyro data acquisition module, an image acquisition and matching module, a global motion estimation module, a gyro residual calculation module, a residual compensation module, an image stabilization compensation module and an image sequence output module, wherein the global motion estimation module is respectively connected to the gyro data acquisition module, the image acquisition and matching module and the gyro residual calculation module, the gyro residual calculation module is respectively connected to the image acquisition and matching module and the residual compensation module, and the image stabilization compensation module is respectively connected to the residual compensation module, the image acquisition and matching module and the image sequence output module; The gyro data acquisition module is used to collect angular velocity data of the gyro sensor and perform data preprocessing; The image acquisition and matching module is used to extract features from the original shaking image sequence, match the position changes of feature points, and estimate local motion parameters between images; The global motion estimation module is used to calculate the global motion parameters of the image sequence according to the preprocessed gyro data; The gyro residual calculation module is used to compare the global motion estimation result based on the gyro with the local motion parameters obtained based on image feature matching, and calculate the residual between the two; The residual compensation module is used to compensate and correct the gyro-based global motion estimation according to the calculated gyro residual; The image stabilization compensation module is used to perform stabilization compensation on the image sequence according to the global motion parameters after residual compensation; The image sequence output module is used to output a stabilized image sequence.

2. The electronic image stabilization system based on gyro residual as claimed in claim 1, characterized in that: The image acquisition and matching module includes an image acquisition unit and an image feature extraction and matching unit. The image acquisition unit is connected to the image feature extraction and matching unit and the global motion estimation module respectively. The image feature extraction and matching unit is connected to the gyro residual calculation module and the image stabilization compensation module respectively.

3. The electronic image stabilization system based on gyro residual as claimed in claim 2, characterized in that: The image acquisition unit is used to acquire the original shaking image sequence and transmit the image data; The image feature extraction and matching unit is used to extract features from the acquired image sequence and match them with feature points in adjacent image frames, and estimate local motion parameters between images by matching position changes of feature points.

4. The electronic image stabilization system based on gyro residual as claimed in claim 1, characterized in that: The gyro data acquisition module comprises a gyro data acquisition unit and a gyro data preprocessing unit, and the gyro data preprocessing unit is connected to the gyro data acquisition unit and the global motion estimation module respectively.

5. The electronic image stabilization system based on gyro residual as claimed in claim 4, characterized in that: The gyro data acquisition unit is used to collect angular velocity data of the gyro sensor; The gyro data preprocessing unit is used to preprocess the collected gyro angular velocity data.

Images