A parameter correction method for gyroscope electronic image stabilization

Abstract

The present invention discloses a parameter correction method for gyroscope electronic anti-shake, and the key points of the technical solution are to obtain a weight matrix K by combining the rotation angle and translation of the X, Y, and Z axes of the camera image sensor acquired by the gyroscope and calculating the rotation angle, lateral translation, and longitudinal translation of the calibration diagram, and complete the parameter correction of the gyroscope electronic anti-shake; obtain an affine matrix P on the basis of the weight matrix K, and perform an inverse affine transformation on the original image of the shaking video through the affine transformation matrix P to obtain an anti-shake image. The correction method of the present invention eliminates the influence of different camera internal parameters on the anti-shake effect, effectively improves the accuracy of the affine transformation matrix P, and improves the effect of the gyroscope electronic anti-shake; effectively reduces or even eliminates the video jitter of the inspection robot in a darker space, and provides help for the post-processing of the video and the viewing of personnel.

Description

Technical Field

[0001] The invention belongs to the technical field of camera video anti-shake, and in particular relates to a parameter correction method for gyroscope electronic anti-shake. Background Art

[0002] Camera video jitter is a common problem in video processing. Video jitter affects the accuracy of camera image acquisition data, and affects data post-processing and viewing by relevant personnel. In order to effectively reduce and eliminate video jitter, camera video anti-shake technology is studied and applied in different fields of image processing such as robot inspection, videography, projection, etc.

[0003] Camera video stabilization is mainly divided into mechanical stabilization and electronic stabilization. Mechanical stabilization has high requirements on external mechanical structure and drive unit, and it is difficult to play its strengths in high-frequency and tiny shakes. Electronic stabilization is divided into image processing method and gyroscope detection and correction method. Image processing method eliminates the subjective "shake" from the perspective of each frame of the video, but it is difficult to apply in low light or when the subject of the video has motion attributes.

[0004] The gyroscope detection and correction method is a video stabilization method that combines the advantages of mechanical stabilization and electronic stabilization. Gyroscope electronic stabilization uses the jitter data of the gyroscope to compensate for the movement and rotation of the video screen. When the gyroscope detection accuracy is sufficient, the compensation weight is the most important parameter for the image stabilization effect. The current mainstream methods for establishing compensation weight parameters are parameter calculation method and trial and error method. Among them, the weight parameter calculation method is determined based on the camera's internal parameters, such as the focal length of the lens, the size of the camera image sensor, the object distance, etc.; another trial and error method is to manually try and finally select a suitable weight parameter based on the subjective jitter of the picture.

[0005] The weight parameter calculation method is determined based on the camera's internal parameters, such as the focal length of the lens, the size of the camera image sensor, the object distance, etc. Its biggest drawback is that it relies on the acquisition of the camera's internal parameters. However, in many cases, the camera's internal parameters cannot be obtained, and the slightest deviation in the internal parameters will affect the anti-shake effect. Another trial-and-error method is to manually try and finally select a suitable weight parameter based on the subjective jitter of the picture. Its disadvantage is that the anti-shake effect depends on the subjective feelings of the personnel. In the face of small-amplitude high-frequency jitter, the trial-and-error method is difficult to achieve good results. The present invention adopts weight parameter optimization based on the picture angle, and obtains the jitter of the video shot by the camera through a machine vision method, which can minimize the jitter of the picture and does not rely on the acquisition of the camera's internal parameters. Summary of the invention

[0006] The object of the present invention is to provide a parameter correction method for gyroscope electronic image stabilization to solve the problems raised in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] The parameter correction method comprises the following steps: collecting the rotation angle and translation of the X, Y and Z axes of the sensor; calibrating the center coordinates and the horizontal line in the calibration diagram; calculating the rotation angle, lateral translation and longitudinal translation of the calibration diagram; combining the rotation angle, lateral translation and longitudinal translation of the calibration diagram with gyroscope data to obtain a weight matrix K to complete parameter correction.

[0009] Preferably, the parameter correction method detects the X, Y, and Z axis rotation angles and translation amounts of the camera image sensor by using a gyroscope, and detects the rotation and translation of the calibration image by using a computer vision method.

[0010] Preferably, the gyroscope is installed in parallel with the camera image sensor, and the central axes of the gyroscope and the camera image sensor are colinear, so as to reduce the coupling between the attitude information of the gyroscope in different directions when the camera shakes significantly.

[0011] Preferably, the parameter correction method uses a closed figure detection method of computer vision to calibrate the coordinates of two circle centers in the calibration graph.

[0012] Preferably, the parameter correction method calculates the angle between the line connecting the two circle centers and the horizontal line to obtain the rotation angle α of the calibration image. img , the rotation angle of the calibration image corresponds to the Z-axis rotation angle α of the camera image sensor sen .

[0013] Preferably, the parameter correction method calculates the coordinates of the midpoint of the line connecting the two circle centers to obtain the lateral translation x of the calibration graph. img , the lateral translation of the calibration image corresponds to the Y-axis rotation angle β of the camera image sensor sen and the X-axis translation x sen .

[0014] Preferably, the parameter correction method calculates the coordinates of the midpoint of the line connecting the two circle centers to obtain the longitudinal translation amount y of the calibration graph. img , the longitudinal translation of the calibration image corresponds to the X-axis rotation angle δ of the camera image sensor sen and Y axis translation y sen .

[0015] Preferably, the parameter correction method combines the transformation information α of the three images img 、x img ,y img and five types of gyroscope data α sen , β sen 、x sen , δ sen ,y senThe weight matrix K is obtained, and the weight matrix is ​​as follows:

[0016] α img =K1α sen

[0017]

[0018]

[0019] Among them, K1, K2, K3, K4, and K5 are the parameters calculated by the camera image sensor's individual Z-axis rotation, Y-axis rotation, X-axis translation, X-axis rotation, and Y-axis translation, respectively.

[0020] Preferably, the parameter correction method obtains an affine matrix P based on the weight matrix K. The affine transformation matrix P performs an inverse affine transformation on the original image of the shaken video to obtain an anti-shake image. The affine matrix P is as follows:

[0021]

[0022] Preferably, the weight matrix K is used for parameter correction of gyro electronic image stabilization. The parameter correction method eliminates the influence of different camera internal parameters on the image stabilization effect, can effectively improve the accuracy of the affine transformation matrix P, and improve the effect of gyro electronic image stabilization.

[0023] Technical effects and advantages of the present invention: The parameter correction method for gyroscope electronic image stabilization of the present invention adopts weight parameter optimization based on the picture angle, and obtains the jitter of the video taken by the camera through a machine vision method, which can minimize the jitter of the picture and does not rely on the acquisition of camera intrinsic parameters. The parameter correction method eliminates the influence of different camera intrinsic parameters on the anti-shake effect, can effectively improve the accuracy of the affine transformation matrix P, and improve the effect of gyroscope electronic image stabilization. The parameter correction method effectively reduces or even eliminates the video jitter of the inspection robot in a darker space, and provides assistance for video post-processing and personnel viewing. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of the parameter correction method for gyroscope electronic anti-shake of the present invention;

[0025] Figure 2 It is a schematic diagram of the structure in which the gyroscope of the present invention is parallel to the camera image sensor;

[0026] Figure 3 It is a schematic diagram of the structure of the present invention containing two white closed circles in a black background;

[0027] Figure 4 It is a structural schematic diagram of the midpoint coordinates of the line connecting the centers of two circles and the rotation angle of the line in the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] The present invention provides a parameter correction method for gyroscope electronic image stabilization, and the parameter correction method comprises the following steps: collecting the rotation angle and translation amount of the X, Y and Z axes of the sensor; calibrating the center coordinates and the horizontal line in the calibration diagram; calculating the rotation angle, the lateral translation amount and the longitudinal translation amount of the calibration diagram; combining the rotation angle, the lateral translation amount and the longitudinal translation amount of the calibration diagram with gyroscope data to obtain a weight matrix K, and completing parameter correction.

[0030] The parameter correction method detects the X-axis, Y-axis, and Z-axis rotation angles and translation amounts of the camera image sensor by using a gyroscope, and detects the rotation and translation of the calibration image by using a computer vision method.

[0031] The gyro electronic image stabilization uses the jitter data of the gyroscope to compensate for the movement and rotation of the video screen. The compensation weight in the jitter data of the gyroscope is the most important parameter for the image stabilization effect when the gyroscope detection accuracy is sufficient. The parameter correction method for the gyroscope electronic image stabilization adopts the weight parameter optimization based on the image angle, and obtains the jitter of the video shot by the camera through the machine vision method, which can minimize the jitter of the image and does not rely on the acquisition of the camera internal parameters.

[0032] The gyroscope is installed in parallel with the camera image sensor, and the central axes of the gyroscope and the camera image sensor are colinear and as close as possible to reduce the coupling between the gyroscope's attitude information in all directions when the camera shakes significantly. The camera image sensor is a camera sensor. The camera image sensor is aligned with the exact center of the captured calibration image to ensure that the camera X and Y axis motions calculated from the video are not coupled with the Z axis rotation.

[0033] The parameter correction method uses a closed figure detection method of computer vision to calibrate the coordinates of two circle centers in the calibration figure.

[0034] The parameter correction method calculates the angle between the line connecting the two circle centers and the horizontal line to obtain the rotation angle α of the calibration image. img , the rotation angle of the calibration image corresponds to the Z-axis rotation angle α of the camera image sensor sen .

[0035] The parameter correction method calculates the coordinates of the midpoint of the line connecting the two circle centers to obtain the lateral translation x of the calibration graph. img , the lateral translation x of the calibration image img Corresponding to the Y-axis rotation angle β of the camera image sensor sen and the X-axis translation x sen , the Y-axis rotation angle of the Sensor is β sen and the X-axis translation x sen The lateral translation x that affects the calibration graph img .

[0036] The parameter correction method calculates the coordinates of the midpoint of the line connecting the two circle centers to obtain the longitudinal translation y of the calibration graph. img , the longitudinal translation y of the calibration image img Corresponding to the X-axis rotation angle δ of the camera image sensor sen and Y axis translation y sen , the X-axis rotation angle of the Sensor is δ sen and Y axis translation y sen Affects the longitudinal translation y of the calibration graph img .

[0037] The parameter correction method combines the transformation information of the three images and five gyroscope data to calculate the correction weight matrix K. The five gyroscope data are the Z-axis rotation angle α of the camera image sensor. sen , Y-axis rotation angle β sen , X-axis rotation angle δ sen , X-axis translation x sen , Y axis translation y sen The three types of image change information are respectively the calibration image rotation angle α img , lateral translation of the calibration graph x img , the longitudinal translation of the calibration image y img The rotation angle of the calibration graph is the rotation angle α of the midpoint coordinates and the connecting line img The lateral translation of the calibration graph is the lateral translation x of the coordinates of the midpoint of the connecting line. img The longitudinal translation of the calibration graph is the longitudinal translation of the coordinates of the midpoint of the connecting line y img , the three types of image change information are camera postures, and the relationship between the camera posture and the image jitter is as follows:

[0038] α img =K1α sen

[0039]

[0040]

[0041] Among them, K1, K2, K3, K4, and K5 are the parameters calculated by the camera image sensor's individual Z-axis rotation, Y-axis rotation, X-axis translation, X-axis rotation, and Y-axis translation, respectively.

[0042] The parameter correction method obtains the X, Y, and Z axis rotation angles of the gyroscope and the X and Y axis translations, and obtains an affine transformation matrix P in combination with the initial weight matrix K. The affine transformation matrix P performs an inverse affine transformation on the original image of the shaking video to obtain an anti-shake image. The affine matrix P is shown as follows:

[0043]

[0044] The weight matrix K is used for parameter correction of gyroscope electronic image stabilization.

[0045] The parameter correction method eliminates the influence of different camera internal parameters on the anti-shake effect, can effectively improve the accuracy of the affine transformation matrix P, and improve the effect of gyroscope electronic anti-shake.

[0046] Furthermore, the calibration diagram is a large blackboard with two white circles.

[0047] Furthermore, the updating of the weight matrix K through the calibration map is an offline process, which is used to calibrate parameters and does not participate in the real-time calculation of the anti-shake process.

[0048] Furthermore, the affine transformation matrix P includes translation and rotation information of the image transformation, but does not include deformation and scaling information.

[0049] Example 1

[0050] The present invention provides a parameter correction method for gyroscope electronic image stabilization. The present invention combines the rotation angle and translation amount of the X, Y, and Z axes of the camera image sensor acquired by the gyroscope and the rotation angle, lateral translation amount, and longitudinal translation amount of the calibration image to obtain a weight matrix K, and completes the parameter correction of the gyroscope electronic image stabilization; based on the weight matrix K, an affine matrix P is obtained, and an anti-shake image is obtained by performing an inverse affine transformation on the original image of the shaking video through the affine transformation matrix P, such as Figure 1 In the process of collecting sensor data, the gyroscope and the camera image sensor are installed in parallel, the central axes of the gyroscope and the camera image sensor are collinear, and the gyroscope and the camera image sensor are as close as possible to ensure that the posture information is not coupled. Figure 2 During the process of collecting sensor data, the camera is aligned with the center of the calibration chart, and the image is ensured to cover all fields of view captured by the camera.

[0051] The calibration image in the calibration process is two closed figures placed horizontally at a certain distance on a pure color background, wherein there is a large brightness difference between the background color and the two closed figures. Figure 3 The black background shown in the figure contains two white closed circles.

[0052] During the calculation process, the calibration image is fixed, the camera simulates a shaking environment to shoot the calibration image, and image processing is performed on each frame of the captured video. During the calculation process, a circle detection method, such as the Hough circle detection method, is used to identify the center coordinates of the two circles in the image, and to calibrate the connecting line and its midpoint; calibrate the horizontal line of the calibration image; calculate the midpoint coordinates of the connecting line and the rotation angle α of the connecting line img ,like Figure 4 The rotation angle of the calibration image corresponds to the Z-axis rotation angle α of the camera image sensor. sen , the horizontal translation x of the coordinates of the midpoint of the connecting line img , the lateral translation of the calibration image corresponds to the Y-axis rotation angle β of the camera image sensor (Sensor) sen and the X-axis translation x sen , the longitudinal translation y of the coordinates of the midpoint of the connecting line img , the longitudinal translation of the calibration image corresponds to the X-axis rotation angle δ of the camera image sensor Sensor sen and Y axis translation y sen .

[0053] While the camera is capturing the calibration image and simulating the shaking, the gyroscope records the data of five postures in real time. sen , β sen 、x sen , δ sen ,y sen , and the above three image posture information α img 、x img ,y img Correspondingly, the proportion value of each corresponding posture constitutes a weight matrix K. The relationship between the image posture and the screen jitter is as follows:

[0054] α img =K1α sen

[0055]

[0056]

[0057] In the above equations, for the lateral and longitudinal translation processes of the calibration image, each equation contains two unknown coefficients (K1, K2) (K3, K4), and each coefficient can be obtained by controlling the irrelevant variables. Control the camera image sensor to rotate only around the Z axis and record αsen and α img , calculate the parameter K1; control the camera image sensor to rotate only around the Y axis and record β sen and x img , calculate the parameter K2; control the camera image sensor to translate only along the X axis, record x sen and x img , calculate the parameter K3; control the camera image sensor to rotate only around the X axis, and record δ sen and img , calculate the parameter K4; control the camera image sensor to translate only along the Y axis and record y sen and img , calculate the parameter K5. So far, the entire coefficient matrix K is established, and the corresponding relationship between the gyroscope data and the image posture information is obtained. The coefficient matrix K obtained above is valid for the camera of this focal length and the object distance condition in this calibration environment.

[0058] In the process of real-time camera anti-shake application, the five posture data α of the gyroscope are sen , β sen 、x sen , δ sen ,y sen Combine the matrix K to calculate the image pose information and synthesize the affine matrix P of the image. The affine matrix P is as follows:

[0059]

[0060] The affine matrix P represents the position relationship between the target image after camera offset and the original image. The original image can be obtained by performing an inverse affine transformation on the offset image. The affine matrix P is established for each frame of the video and an inverse affine transformation is performed to obtain a video with the jitter removed.

[0061] Working principle: The parameter correction method of the gyroscope electronic image stabilization of the present invention uses a camera to shoot a calibration image and simulate the shaking process. The gyroscope records five posture data in real time and corresponds to three image posture information. The five posture data are the Z-axis rotation angle α of the camera image sensor Sensor. sen , Y-axis rotation angle β sen , X-axis rotation angle δ sen , X-axis translation x sen , Y axis translation y sen The proportion values ​​corresponding to the three image postures constitute the weight matrix K, and the three image postures are the rotation angles α of the calibration image. img , lateral translation of the calibration graph x img and the longitudinal translation y of the calibration graph img The rotation angle of the calibration graph is the rotation angle α of the midpoint coordinates and the connecting lineimg , the calibration image is rotated by an angle α img Corresponding to the Z-axis rotation angle α of the camera image sensor Sensor sen The lateral translation of the calibration graph is the lateral translation x of the coordinates of the midpoint of the connecting line. img , the lateral translation x of the calibration image img Corresponding to the Y-axis rotation angle β of the camera image sensor Sensor sen and the X-axis translation x sen The longitudinal translation of the calibration graph is the longitudinal translation of the coordinates of the midpoint of the connecting line y img , the longitudinal translation y of the calibration image img Corresponding to the X-axis rotation angle δ of the camera image sensor Sensor sen and Y axis translation y sen The weight matrix K is combined with each frame of gyroscope data to establish an affine matrix P, and the affine matrix P performs an inverse affine transformation on the current frame to obtain the final anti-shake image.

[0062] The weight matrix K is as follows:

[0063] α img =K1α sen

[0064]

[0065]

[0066] The affine matrix P is as follows:

[0067]

[0068] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A parameter correction method for gyroscope electronic image stabilization, characterized in that: The parameter correction method comprises the following steps: The X, Y, and Z axis rotation angles and translations of the camera image sensor are detected by a gyroscope, and the rotation and translation of the calibration image are detected by a computer vision method. Use the closed figure detection method of computer vision to calibrate the coordinates of the two center points in the calibration figure, and calibrate the horizontal line of the calibration figure; Calculate the angle between the line connecting the two circle centers and the horizontal line to get the rotation angle α of the calibration graph img , the rotation angle of the calibration image corresponds to the Z-axis rotation angle α of the camera image sensor sen ; Calculate the coordinates of the midpoint of the line connecting the two circle centers to obtain the lateral translation x of the calibration graph img , the lateral translation of the calibration image corresponds to the Y-axis rotation angle β of the camera image sensor sen and the X-axis translation x sen ; Calculate the coordinates of the midpoint of the line connecting the two circle centers to obtain the longitudinal translation y of the calibration graph img , the longitudinal translation of the calibration image corresponds to the X-axis rotation angle δ of the camera image sensor sen and Y axis translation y sen ; Combining the transformation information of the three images α img 、x img ,y img and five types of gyroscope data α sen , β sen 、x sen , δ sen ,y sen The weight matrix K is obtained, and the weight matrix K is as follows: α img ＝K1α sen Among them, K1, K2, K3, K4, and K5 are the parameters calculated by the camera image sensor's individual Z-axis rotation, Y-axis rotation, X-axis translation, X-axis rotation, and Y-axis translation, respectively; Based on the weight matrix K, an affine matrix P is obtained. The affine matrix P performs an inverse affine transformation on the original image of the shaking video to obtain an anti-shake image. The affine matrix P is as follows:

2. The parameter correction method for gyro electronic image stabilization according to claim 1, characterized in that: The gyroscope is installed in parallel with the camera image sensor, and the central axes of the gyroscope and the camera image sensor are colinear.

3. The parameter correction method for gyro electronic image stabilization according to claim 1, characterized in that: The weight matrix K is used for parameter correction of gyroscope electronic image stabilization.

4. The parameter correction method for gyro electronic image stabilization according to any one of claims 1 to 3, characterized in that: The parameter correction method eliminates the influence of different camera internal parameters on the anti-shake effect.

Images