Distortion Correction Method Based on Ray-pixel Camera Calibration Model

Abstract

The present invention discloses a distortion correction method based on a Ray-pixel camera calibration model, including calibrating the camera using the Ray-pixel model to obtain the camera intrinsics; referring to the focal length (f x , f y ) and the principal point coordinates (c x , c y ) of the lens, and selecting a perspective transformation model; using the selected perspective transformation model to back-project the pixel (u, v) to obtain the incident light direction corresponding to the pixel #imgabs0# re-projecting the back-projected direction using the Raxel-pixel model to obtain p'=(u', u'); performing bilinear interpolation on p' on the original distorted image to obtain the color of p', and assigning this color to the pixel p, and finally obtaining the corrected undistorted image. The image content after distortion correction in the present invention can more truly reflect the appearance of the object and is also more suitable for further applying algorithms such as image processing, object detection, and image segmentation.

Description

Technical Field

[0001] The present invention relates to the technical field of camera calibration and lens optical imaging, and in particular to a distortion correction method based on a Ray-pixel camera calibration model. Background Art

[0002] Camera calibration and distortion correction are widely used in embedded cameras, robot vision, autonomous driving and other fields. By calibrating and dedistorting the original image, the distortion in the input image can be effectively removed, so that the distorted image can more realistically reflect the appearance of the object. Different lenses have different types of distortion. For example, the general medium and long focal length lens has very little distortion, and better distortion correction can be achieved using simple radial distortion and tangential distortion coefficients.

[0003] However, some commonly used short-focus lenses in the industry, such as car lenses or fisheye lenses, have large distortions. Due to the limitations of the lens production process, simple radial distortion and tangential distortion cannot accurately describe the actual distortion of the lens. Therefore, a camera calibration and distortion correction method with wider applicability and more complex parameters is needed. In 2001, Michael D. Grossberg proposed a universal camera model based on Ray-pixel, which regards the imaging system as consisting of multiple basic units of "photosensitive elements-incident light". This model more intuitively describes the imaging process of objects in the camera at the physical level, regardless of the type of lens and the shortcomings of the manufacturing process, and is applicable to all types of imaging systems.

[0004] Thomas Schops also published the algorithm details and implementation of the calibration model in 2020. However, there is still no distortion correction method based on the Ray-pixel model, which makes the model very inconvenient to apply. Summary of the invention

[0005] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is to provide a distortion correction method based on a Ray-pixel camera calibration model to solve the deficiencies of the prior art.

[0006] To achieve the above object, the present invention provides a distortion correction method based on a Ray-pixel camera calibration model, comprising the following steps:

[0007] Step 1: Use the Ray-pixel model to calibrate the camera and obtain the camera's intrinsic parameters Intrinsics;

[0008] Step 2: Reference lens focal length (f x , f y ), principal point coordinates (c x, c y ), select a perspective transformation model;

[0009] Step 3: Use the perspective transformation model selected in step 2 to back-project the pixel (u, v) to obtain the incident light direction corresponding to the pixel.

[0010] Step 4: Reproject the back-projected direction using the Raxel-pixel model to obtain p′=(u′,u′);

[0011] Step 5: Perform bilinear interpolation on p' on the distorted original image to obtain the color of p', assign the color to pixel p, and finally obtain the corrected dedistorted image.

[0012] Furthermore, the camera's intrinsic parameter Intrinsics is a three-dimensional vector (d x , d y , d z ), which indicates the direction of the incident light corresponding to the pixel (x, y).

[0013] Furthermore, the focal length (f x , f y ), principal point coordinates (c x , c y ),satisfy:

[0014] f x =f y =f / k, where f is the focal length of the lens, k is the physical side length of each pixel of the sensor, and f x and f y is the focal length expressed in pixels;

[0015] c x =IMAGE_W / 2, where IMAGE_W is the width of the input image (in pixels);

[0016] c y =IMAGE_H / 2, where IMAGE_H is the height of the input image (in pixels).

[0017] Furthermore, when the perspective transformation model in step 3 performs back-projection on the pixel (u, v), a three-dimensional direction vector is obtained. The three-dimensional direction vector The modulus length is arbitrary, and the z-axis component of the direction vector is uniformly fixed to 1, that is, (X c , Y c , 1), the back projection can be expressed by the inverse matrix of A:

[0018] [Xc ; Y c ; Z c ]=A -1 s[u;v;1],X c =1

[0019] X c =(uc x ) / f x

[0020] Y c =(vc y ) / f y

[0021] Therefore, the direction of the incident light obtained by back-projection is

[0022] Furthermore, the step 4 uses the Raxel-pixel model to reproject the back-projected direction, specifically:

[0023] First, initialize the position of the projection point, which is generally initialized to the center point of the image, that is, (IMAGE_W / 2,IMAGE_H / 2);

[0024] Next, the Levenberg-Marquardt algorithm is used to iteratively optimize the position of the projection point so that the back-projection direction of the projection point continuously approaches the direction of the incident light obtained by back-projection. When the target loss is lower than a certain threshold, the iteration ends; wherein the back-projection direction is obtained by B-Spline plane interpolation.

[0025] Furthermore, the color of p' in step 4 is in RGB format or grayscale format.

[0026] The beneficial effects of the present invention are:

[0027] 1. Since the parameters of the Ray-pixel model used are more intuitive and the parameter quantity is sufficient to describe the actual distortion of the lens, combined with the distortion correction method proposed in the present invention, it is possible to accurately correct the distortion of lenses with different degrees of distortion, including medium-to-long focal length lenses, short focal length lenses, fisheye lenses, etc.

[0028] 2. When performing distortion correction on images with large distortion, compared with the distortion correction of the pinhole model (the average reprojection error is about 0.3 pixels), the distortion correction method proposed in the present invention can achieve a smaller reprojection error, and the average reprojection error of the image can be controlled within 0.1 pixels.

[0029] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a structural schematic diagram of the present invention.

[0031] Figure 2 It is a principle block diagram of the electromagnetic size control circuit of the present invention.

[0032] Figure 3 It is the principle diagram of the lifting control circuit of the present invention. DETAILED DESCRIPTION

[0033] Example 1

[0034] like Figure 1 As shown, the present invention provides a distortion correction method based on a Ray-pixel camera calibration model, comprising the following steps:

[0035] Step 1: Use the Ray-pixel model to calibrate the camera and obtain the camera's intrinsic parameters Intrinsics;

[0036] Step 2: Reference lens focal length (f x , f y ), principal point coordinates (c x , c y ), select a perspective transformation model;

[0037] Step 3: Use the perspective transformation model selected in step 2 to back-project the pixel (u, v) to obtain the incident light direction corresponding to the pixel.

[0038] Step 4: Reproject the back-projected direction using the Raxel-pixel model to obtain p′=(u′,u′);

[0039] Step 5: Perform bilinear interpolation on p' on the distorted original image to obtain the color of p', assign the color to pixel p, and finally obtain the corrected dedistorted image.

[0040] The following is a description of the above steps:

[0041] 1. Internal parameter form of Ray-pixel model

[0042] There are a total of IMAGE_W*IMAGE_H vectors, each of which has the following shape: (d x , d y , d z ) represents the normalized direction of the incident light.

[0043] 2. Selection of perspective model

[0044] You can refer to the parameters and specifications of the camera lens and sensor to select:

[0045] f x =f y =f / k. Where f is the focal length of the lens (in mm); k is the physical side length of each pixel of the sensor (in mm). Therefore, f x and f y is the focal length in pixels;

[0046] c x =IMAGE_W / 2. IMAGE_W is the width of the input image (in pixels).

[0047] c y =IMAGE_H / 2, where IMAGE_H is the height of the input image (in pixels).

[0048] The perspective transformation matrix corresponding to the perspective model is:

[0049]

[0050] 3. Un-projection

[0051] When the perspective transformation model (pinhole model) in step 3 back-projects the pixel (u, v), a three-dimensional direction vector is obtained with an arbitrary modulus length. Here, the z-axis component of the direction vector is uniformly fixed to 1, that is, (X c , Y c , 1), the back projection can be expressed by the inverse matrix of A:

[0052] [X c ; Y c ; Z c ]=A -1 s[u;v;1],X c =1

[0053] X c =(uc x ) / f x

[0054] Y c =(vc y ) / f y

[0055] Therefore, the direction of the incident light obtained by back-projection is

[0056] 4. Projection

[0057] The ray-pixel projection process is relatively complex and is an optimization process:

[0058] First, initialize the position of the projection point, which is generally initialized to the center point of the image, that is, (IMAGE_W / 2, IMAGE_H / 2);

[0059] Next, the Levenberg-Marquardt algorithm is used to iteratively optimize the position of the projection point so that the back-projection direction of the projection point (obtained by B-Spline plane interpolation) continuously approaches When the target loss is below a certain threshold, the iteration ends.

[0060] 5.Cubic B-Spline Planar Interpolation

[0061] In the iterative process of the Levenberg-Marquardt algorithm, the coordinates of the projection point p″ are not integer type, and the internal parameters of Ray-pixel are pixel-by-pixel (sparse), so it is necessary to obtain the back-projection direction of the coordinates between adjacent pixels (at the pixel level) by interpolation. (i.e. the direction of the incident light).

[0062] Assume that the four pixels surrounding the projection point p″ are p″ 1 , p″ 2 , p″ 3 , p″ 4 , and their corresponding incident light directions are (Get it directly from the Ray-pixel internal parameters). At this time, the four direction vectors are interpolated by Cubic B-Spline plane to calculate The form of CubicB-Spline plane interpolation is:

[0063]

[0064] Among them, m and n are both 1, and the orders p and q are both 3.

[0065] 6. Bilinear interpolation

[0066] The projection point p″=(x″, y″) obtained by the Raxel-pixel model is still at the sub-pixel level, and the color g″ of the point also needs to be determined by bilinear interpolation.

[0067] Assume that the four pixels surrounding the projection point p″ (in the original image) are

[0068] p″1，1 =(x 1 ,y 1 ), p″ 2，1 =(x 2 ,y 1 ), P″ 1，2 =(x 1 ,y 2 ), p″ 2，2 =(x 2 ,y 2 ),

[0069] Their corresponding pixel grayscale values ​​are g 1，2 , g 2，1 , g 1，2 , g 2，2 , then:

[0070] w 1 =(x 2 -x″) / (x 2 -x 1 )

[0071] w 2 =(x″-x 1 ) / (x 2 -x 1 )

[0072] w 3 =(y 2 -y″) / (y 2 -y 1 )

[0073] w 4 =(y″-y 1 ) / (y 2 -y 1 )

[0074] g″=w 3 *(g 1，1 *w 1 +g 2，1 *w 2 )+w 4 *(g 2，1 *w 1 +g 2，2 *w 2 )

[0075] Example 2

[0076] like Figure 1 As shown, this embodiment is a distortion correction method based on the Ray-pixel model, comprising the steps of:

[0077] 1. Use a camera to take a set of pictures of the calibration corner points. Specific details: Fix the camera on a tripod and keep it still throughout the process, and adjust the ambient light to ensure normal brightness and illumination. The experimenter holds a chessboard (0.5m to 1m away from the camera) so that the chessboard appears within the camera's field of view to take pictures, and then slightly adjust the angle and distance of the chessboard to take about 15 pictures. Then use OpenCV's cv::findChessboardCorners method to detect corners in all pictures and obtain the corner data required for calibration.

[0078] 2. Use the fisheye pinhole model provided by OpenCV for calibration, count the reprojection error and perform distortion correction. Specific details:

[0079] Use OpenCV's cv::fisheye::calibrate method to calibrate the focus data and obtain the camera's intrinsic parameter matrix and distortion matrix. Here, radial distortion k1, k2, k3, and k4 are used. The reprojection error obtained by this function is 0.43 (in pixels). Figure 2 It can be seen that the correction effect in some areas of the edge of the image is obviously not ideal.

[0080] 3. Use Ray-pixel to calibrate, calculate the reprojection error and perform distortion correction. The reprojection error obtained by the calibration method is 0.092 (unit pixel).

[0081] Repeating the above experiment three times on the same camera and on three different cameras can obtain similar results. Note: The camera lens focal length used in this experiment is 2.4mm and the image resolution is 1280x960.

[0082] Through the above experiments, it can be clearly found that the Ray-pixel camera calibration and distortion correction method has a better dedistortion effect on the image (both the reprojection error and the distortion correction effect diagram can be reflected).

[0083] The preferred specific embodiments of the present invention are described in detail above. A person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. Distortion correction method based on Ray-pixel camera calibration model, It is characterized in that The following steps are involved: Step 1: Use the Ray-pixel model to calibrate the camera and obtain the camera's intrinsic parameters Intrinsics; Step 2: Reference lens focal length (f x , f y ), principal point coordinates (c x , c y ), select a perspective transformation model; Step 3: Use the perspective transformation model selected in step 2 to back-project the pixel (u, v) to obtain the incident light direction corresponding to the pixel. Step 4: Use the Rayxel-pixel model to reproject the back-projected direction to obtain p'=(u',v'); Step 5: Perform bilinear interpolation on p' on the distorted original image to obtain the color of p', assign the color to pixel p, and finally obtain the corrected dedistorted image; The camera's intrinsic parameter Intrinsics is a three-dimensional vector (d x ,d y ,d z ), which indicates the direction of the incident light corresponding to the pixel (x, y); The focal length (f x , f y ), principal point coordinates (c x , c y ),satisfy: f x =f y =f / k, where f is the focal length of the lens, k is the physical side length of each pixel of the sensor, and f x and f y is the focal length expressed in pixels; c x =IMAGE_W / 2, where IMAGE_W is the width of the input image in pixels; c y =IMAGE_H / 2, where IMAGE_H is the height of the input image, in pixels.

2. The distortion correction method based on the Ray-pixel camera calibration model as claimed in claim 1, It is characterized in that When the perspective transformation model in step 3 back-projects the pixel (u, v), a three-dimensional direction vector is obtained. The three-dimensional direction vector has an arbitrary modulus length. The z-axis component of this direction vector is uniformly fixed to 1, i.e., (X c , Y c , 1), and this back-projection is represented by the inverse matrix of A: (X c ,Y c ,Z c )=A -1 s[u;v;1],X c =1 X c =(u-c x ) / f x Y c =(v-c x ) / f y Therefore, the direction of the incident light obtained by back-projection is 3. The distortion correction method based on the Ray-pixel camera calibration model as claimed in claim 1, It is characterized in that The step 4 uses the Ray-pixel model to reproject the direction after back-projection, specifically: First, initialize the position of the projection point, which is generally initialized to the center point of the image, that is, (IMAGE_W / 2, IMAGE_H / 2); Next, the Levenberg-Marquardt algorithm is used to iteratively optimize the position of the projection point so that the back-projection direction of the projection point continuously approaches the direction of the incident light obtained by back-projection. When the target loss is lower than a certain threshold, the iteration ends; wherein the back-projection direction is obtained by B-Spline plane interpolation.

4. The distortion correction method based on the Ray-pixel camera calibration model according to claim 1, It is characterized in that In step 4, the color of p' is in RGB format or grayscale format.

Images