Image correction method and system for rolling shutter image

Abstract

The invention relates to an image correction method and system for a rolling shutter image, and the method comprises the following steps: obtaining a scene image through a rolling shutter of a camera, and obtaining equipment posture data at different moments in real time through an IMU; calculating an equipment attitude angle (p, y, r) at each equipment attitude data acquisition moment after the acquisition moment t1 of the first equipment attitude data, and calculating a correction matrix KIMU based on the equipment attitude angles (p, y, r); and correcting the image line corresponding to the correction matrix KIMU according to the correction matrix KIMU to obtain a corrected image line. According to the method, the equipment attitude data at different moments are obtained based on the IMU, the correction matrixes KIMU at different moments are constructed, then the corresponding image lines are independently corrected based on the correction matrixes KIMU, and finally the image lines of the whole scene image are corrected through interpolation operation.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, and in particular to an image correction method and system for rolling shutter images. Background Technology

[0002] In existing technologies, rolling shutters can be mounted on mobile devices (such as drones, unmanned vehicles, robots, and manned vehicles) to complete imaging. They have a significant cost advantage over global shutters and can significantly improve image quality in low-light environments.

[0003] However, because rolling shutter imaging uses line-by-line exposure, image distortion can occur when the movable device is shaken (e.g., ...). Figure 1 (As shown). Therefore, there is an urgent need for a technical solution to improve the problem of image distortion caused by shaking during rolling shutter photography. Summary of the Invention

[0004] The purpose of this invention is to provide an image correction method and system for rolling shutter images, which acquires device attitude data at different times based on an IMU and constructs a correction matrix K for different times. IMU Then based on the correction matrix K IMU The corresponding image rows are corrected individually, and finally the image rows of the entire scene image are corrected through interpolation.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] On the one hand, an image correction method for rolling shutter images is provided, which includes the following steps:

[0007] Scene images are acquired using the camera's rolling shutter, and simultaneously, device attitude data (g_x, g_y, g_z, t) is acquired in real time at different times using the IMU. Here, the IMU coordinate system is Os-XsYsZs, t is the acquisition time of the device attitude data, g_x represents the angular velocity of the IMU coordinate system when it rotates by an angle p around the Xs axis, g_y represents the angular velocity of the IMU coordinate system when it rotates by an angle y around the Ys axis, and g_z represents the angular velocity of the IMU coordinate system when it rotates by an angle r around the Zs axis.

[0008] After calculating the acquisition time t1 of the first device attitude data, calculate the device attitude angle (p, y, r) at each device attitude data acquisition time, and calculate the correction matrix K based on the device attitude angle (p, y, r). IMU；

[0009] According to the correction matrix K IMU For the correction matrix KIMU The corresponding image rows are corrected to obtain the corrected image rows;

[0010] In addition, interpolation is performed based on the corrected image rows to correct the entire scene image acquired by the rolling shutter.

[0011] Preferably, the device attitude angle (p, y, r) is calculated according to the following formula: .

[0012] Preferably, the correction matrix K is calculated according to the following formula. IMU :

[0013]

[0014] Where K is the camera intrinsic parameter matrix; R is the camera rotation matrix.

[0015] Preferably, the corrected image rows are obtained using the following formula:

[0016]

[0017] Among them, (x n y n (t) represents time t before correction. n Collected device attitude data (g_x) n , g_y n g_z n , t n The pixel coordinates of the image row corresponding to (u) n v n The corrected device attitude data (g_x) n , g_y n g_z n , t n The pixel coordinates of the image row corresponding to ).

[0018] Preferably, interpolation is performed based on the following formula:

[0019]

[0020] Where k is the number of image rows between two adjacent device pose data, and k = h / n, h is the height of the entire scene image acquired by the rolling shutter; j represents the j-th image row; , () represents the pixel coordinates in the corrected scene image.

[0021] Preferably, the image correction method further includes the following steps:

[0022] The pixels of the corrected scene image are corrected to obtain the corrected pixel coordinates of each pixel.

[0023] Preferably, the pixels of the corrected scene image are corrected to obtain the corrected pixel coordinates of each pixel, which includes the following steps:

[0024] Define a sliding window;

[0025] Move the sliding window to the current pixel coordinates ( , ) as the center of 9 9. Slide within the neighborhood and calculate the sum of the differences in grayscale values ​​of the pixels after each slide using the following formula: sum m,n :

[0026]

[0027] Where I is the gray value of the pixel;

[0028] Sum of the differences in grayscale values m,n The smallest (m,n) is determined as the current pixel coordinate. , Corrected pixel coordinates;

[0029] Repeat the above steps to obtain the corrected pixel coordinates of each pixel in the entire scene image after each correction.

[0030] Preferably, each row of images exposed by the rolling shutter corresponds uniquely to a device attitude data.

[0031] Preferably, the camera and IMU are both mounted on a mobile device.

[0032] On the other hand, an image correction system for rolling shutter images is also provided, comprising:

[0033] IMU, which is used to acquire device pose data (g_x, g_y, g_z, t) at different times in real time while acquiring scene images.

[0034] The attitude angle calculation module is used to calculate the device attitude angle (p, y, r) at each device attitude data acquisition time after the acquisition time t1 of the first device attitude data.

[0035] The correction matrix calculation module is used to calculate the correction matrix K based on the device attitude angles (p, y, r). IMU ;

[0036] Image line correction module, which is used to correct the image based on the correction matrix K IMU For the correction matrix K IMU The corresponding image rows are corrected to obtain the corrected image rows;

[0037] And an interpolation calculation module, which performs interpolation calculations based on the corrected image rows to correct the entire scene image acquired by the rolling shutter.

[0038] In summary, the present invention has the following advantages compared with the prior art:

[0039] This invention acquires device attitude data at different times based on IMU and constructs a correction matrix K at different times. IMU Then based on the correction matrix K IMU The corresponding image rows are corrected individually, and finally the image rows of the entire scene image are corrected through interpolation. Furthermore, based on the sliding window and the difference in gray values ​​of pixels, the corrected pixel coordinates of each pixel are determined to achieve further accurate correction of the scene image. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of a distorted image;

[0041] Figure 2 This is a flowchart of the image correction method in this invention;

[0042] Figure 3 This is a schematic diagram showing the positional relationship between the camera coordinate system and the IMU coordinate system in this invention;

[0043] Figure 4 This is a schematic diagram illustrating the correspondence between device posture data and image rows in this invention;

[0044] Figure 5 This is a schematic diagram illustrating the interpolation calculation process in this invention;

[0045] Figure 6 In this invention, the sliding window is at 9 A schematic diagram of sliding within a 9-neighborhood;

[0046] Figure 7 This is a schematic diagram of the image correction system in this invention. Detailed Implementation

[0047] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0048] Example 1:

[0049] like Figure 2As shown, this embodiment provides an image correction method for rolling shutter images, which includes the following steps:

[0050] S1, such as Figure 3 As shown, the camera coordinate system is defined as Oc-XcYcZc and the IMU coordinate system is defined as Os-XsYsZs, with the Xc and Xs axes parallel to each other, the Yc and Ys axes parallel to each other, and the Zc and Zs axes parallel to each other.

[0051] While acquiring scene images through the rolling shutter of the camera on the mobile device, the device's attitude data (g_x, g_y, g_z, t) is acquired in real time at different moments in chronological order through the IMU (Inertial Measurement Unit) on the mobile device. Here, t is the acquisition time of the device attitude data, g_x represents the angular velocity of the IMU coordinate system rotating around the Xs axis by an angle p when the mobile device rotates around Xc and performs a pitching motion, g_y represents the angular velocity of the IMU coordinate system rotating around the Ys axis by an angle y when the mobile device rotates around Yc and performs a yaw motion, and g_z represents the angular velocity of the IMU coordinate system rotating around the Zs axis by an angle r when the mobile device rotates around Zc and performs a rolling motion.

[0052] In this embodiment, as Figure 4 As shown, the IMU operates at times t1, t2, ..., t... n The corresponding acquired device attitude data are (g_x1, g_y1, g_z1, t1), (g_x2, g_y2, g_z2, t2), ..., (g_x n , g_y n g_z n , t n ), where n is a positive integer greater than or equal to 2, and each device attitude data corresponds to an image row exposed by a rolling shutter.

[0053] Preferably, in this embodiment, each row of images exposed by the rolling shutter uniquely corresponds to a device attitude data point to enhance the accuracy of subsequent image correction; for example, such as Figure 3 As shown, there are a total of n rows of images exposed by the rolling shutter. The device attitude data are (g_x1, g_y1, g_z1, t1), (g_x2, g_y2, g_z2, t2), ..., (g_x... n , g_y n g_z n , t n Each of these corresponds one-to-one with the 1st, 2nd, ..., nth image row;

[0054] S2. After calculating the acquisition time t1 of the first device attitude data according to the following formula (1), calculate the device attitude angle (p, y, r) and correction matrix K at each device attitude data acquisition time. IMU ;

[0055]

[0056]

[0057] Where K is the camera intrinsic parameter matrix, which can be obtained by calibrating the camera parameters; R is the camera rotation matrix; in this embodiment, both the camera intrinsic parameter matrix K and the camera rotation matrix R can be determined using existing technologies, and will not be elaborated further; K IMUn Let t be the time of the nth device attitude data acquisition. n The correction matrix;

[0058] S3, based on the correction matrix K IMU For the correction matrix K IMU The corresponding image rows are corrected to obtain corrected image rows. Specifically, in this embodiment, the corrected image rows are obtained using the following formula:

[0059]

[0060] Among them, (x n y n (t) represents time t before correction. n Collected device attitude data (g_x) n , g_y n g_z n , t n The pixel coordinates of the image row corresponding to (u) n v n The corrected device attitude data (g_x) n , g_y n g_z n , t n The pixel coordinates of the image row corresponding to ).

[0061] And S4, perform interpolation operations based on the corrected image rows to correct the entire scene image acquired by the rolling shutter;

[0062] Specifically, such as Figure 5 As shown, in this embodiment, bilinear interpolation is used to process the image rows to complete the distortion correction of the entire scene image. This operation can be implemented using the following formula:

[0063]

[0064] Where k is the number of image rows between two adjacent device pose data, and k = h / n, h is the height of the entire scene image acquired by the rolling shutter, such as when h = 2160 and n = 100, k = 21.6; j represents the j-th image row; , () represents the pixel coordinates in the corrected scene image.

[0065] Therefore, this embodiment can first acquire device attitude data at different times based on the IMU, and then construct the correction matrix K at different times. IMU Then based on the correction matrix K IMU The corresponding image rows are corrected individually, and finally, the entire scene image is corrected through interpolation to improve the distortion of the scene image (i.e., restore it to its original state). Figure 1 The effect shown in the image on the left.

[0066] Example 2:

[0067] The only difference between this embodiment and Embodiment 1 is that, to further ensure the image distortion correction effect, the image correction method further includes the following steps:

[0068] S5. Correct the pixels of the corrected scene image to obtain the corrected pixel coordinates of each pixel. This includes the following steps:

[0069] Define a 3x3 matrix centered at any pixel (x, y) in the corrected scene image. A 3-pixel area is used as a sliding window;

[0070] Move the sliding window to the current pixel coordinates ( , ) as the center of 9 9. Slide within the neighborhood and calculate the sum of the differences in grayscale values ​​of the pixels after each slide using the following formula: sum m,n :

[0071]

[0072] Where I is the gray value of the pixel;

[0073] Sum of the differences in grayscale values m,n The smallest (m,n) is determined as the current pixel coordinate. , Corrected pixel coordinates;

[0074] Repeat the above steps to obtain the corrected pixel coordinates of each pixel in the entire scene image after each correction.

[0075] Therefore, in this embodiment, the difference between the sliding window and the pixel grayscale value is used to further determine the coordinates of the current pixel. , Corresponding to ).

[0076] Example 3:

[0077] This embodiment discloses an image correction system for rolling shutter images, which can implement the image correction method described in Embodiment 1 or 2, such as... Figure 7 As shown, the image correction system includes:

[0078] IMU 2, which is mounted on a mobile device, is used to acquire device attitude data (g_x, g_y, g_z, t) at different times in real time while acquiring scene images. The process is the same as step S1. The scene images are acquired based on camera 1, which is mounted on a mobile device and has a rolling shutter.

[0079] The attitude angle calculation module 3 is used to calculate the device attitude angle (p, y, r) at each device attitude data acquisition time after the acquisition time t1 of the first device attitude data. Its process is the same as step S2.

[0080] Correction matrix calculation module 4 is used to calculate correction matrix K based on the device attitude angles (p, y, r). IMU The process is the same as step S2;

[0081] Image line correction module 5, which is used to correct the image line correction matrix K. IMU For the correction matrix K IMU The corresponding image rows are corrected to obtain corrected image rows, and the process is the same as step S3.

[0082] Interpolation calculation module 6 is used to perform interpolation calculations based on the corrected image rows to correct the entire scene image acquired by the rolling shutter. The process is the same as step S4.

[0083] And, pixel correction module 7, which is used to adjust the sliding window at the current pixel coordinates ( , ) as the center of 9 9. Slide within the neighborhood, and calculate the sum of the differences in grayscale values ​​of the pixels after each slide. m,n And the sum of the differences in grayscale values. m,n The smallest (m,n) is determined as the current pixel coordinate. , The process of correcting pixel coordinates is the same as in Example 2.

[0084] In summary, this invention can acquire device attitude data at different times based on the IMU and construct correction matrices K at different times. IMU Then based on the correction matrix K IMU The corresponding image rows are corrected individually, and finally the image rows of the entire scene image are corrected through interpolation. Furthermore, based on the sliding window and the difference in gray values ​​of pixels, the corrected pixel coordinates of each pixel are determined to achieve further precise correction of the scene image, thereby improving the distortion of the scene image.

[0085] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image correction method for images captured by a rolling shutter, characterized in that, Includes the following steps: Scene images are acquired using the camera's rolling shutter, and simultaneously, device attitude data (g_x, g_y, g_z, t) is acquired in real time at different times using the IMU. Here, the IMU coordinate system is Os-XsYsZs, t is the acquisition time of the device attitude data, g_x represents the angular velocity of the IMU coordinate system when it rotates by an angle p around the Xs axis, g_y represents the angular velocity of the IMU coordinate system when it rotates by an angle y around the Ys axis, and g_z represents the angular velocity of the IMU coordinate system when it rotates by an angle r around the Zs axis. After calculating the acquisition time t1 of the first device attitude data, calculate the device attitude angle (p, y, r) at each device attitude data acquisition time, and calculate the correction matrix K based on the device attitude angle (p, y, r). IMU； According to the correction matrix K IMU For the correction matrix K IMU The corresponding image rows are corrected to obtain the corrected image rows; In addition, interpolation is performed based on the corrected image rows to correct the entire scene image acquired by the rolling shutter.

2. The image correction method as described in claim 1, characterized in that, Calculate the device attitude angles (p, y, r) using the following formula: .

3. The image correction method as described in claim 2, characterized in that, The correction matrix K is calculated using the following formula. IMU : ； Where K is the camera intrinsic parameter matrix; R is the camera rotation matrix.

4. The image correction method as described in claim 3, characterized in that, The corrected image rows are obtained using the following formula: ； Among them, (x n y n (t) represents time t before correction. n Collected device attitude data (g_x) n , g_y n g_z n , t n The pixel coordinates of the image row corresponding to (u) n v n The corrected device attitude data (g_x) n , g_y n g_z n , t n The pixel coordinates of the image row corresponding to ).

5. The image correction method as described in claim 1, characterized in that, Interpolation is performed based on the following formula: ； Where k is the number of image rows between two adjacent device pose data, and k = h / n, h is the height of the entire scene image acquired by the rolling shutter; j represents the j-th image row; , () represents the pixel coordinates in the corrected scene image.

6. The image correction method as described in claim 1, characterized in that, The image correction method further includes the following steps: The pixels of the corrected scene image are corrected to obtain the corrected pixel coordinates of each pixel.

7. The image correction method as described in claim 6, characterized in that, The pixels of the corrected scene image are corrected to obtain the corrected pixel coordinates of each pixel, which includes the following steps: Define a sliding window; Move the sliding window to the current pixel coordinates ( , ) as the center of 9 9. Slide within the neighborhood and calculate the sum of the differences in grayscale values ​​of the pixels after each slide using the following formula: sum m,n : ； Where I is the gray value of the pixel; Sum of the differences in grayscale values m,n The smallest (m,n) is determined as the current pixel coordinate. , Corrected pixel coordinates; Repeat the above steps to obtain the corrected pixel coordinates of each pixel in the entire scene image after each correction.

8. The image correction method as described in claim 1, characterized in that, Each row of images exposed by the rolling shutter is uniquely associated with a device attitude data.

9. The image correction method as described in claim 1, characterized in that, Both the camera and IMU are mounted on a mobile device.

10. An image correction system for rolling shutter images, characterized in that, include: IMU, which is used to acquire device pose data (g_x, g_y, g_z, t) at different times in real time while acquiring scene images. The attitude angle calculation module is used to calculate the device attitude angle (p, y, r) at each device attitude data acquisition time after the acquisition time t1 of the first device attitude data. The correction matrix calculation module is used to calculate the correction matrix K based on the device attitude angles (p, y, r). IMU ; Image line correction module, which is used to correct the image based on the correction matrix K IMU For the correction matrix K IMU The corresponding image rows are corrected to obtain the corrected image rows; And an interpolation calculation module, which performs interpolation calculations based on the corrected image rows to correct the entire scene image acquired by the rolling shutter.

Images