A surround view system image correction method

Abstract

The present invention provides a method for correcting images of a surround view system, the method comprising: step 1, receiving an image to be calibrated and a calibration auxiliary image, the image to be calibrated is an image taken by a fisheye camera of the surround view system and after distortion correction, and the calibration auxiliary image is an undistorted image taken by the calibration camera; step 2, aligning the image to be calibrated and the calibration auxiliary image to obtain an overlapping area; step 3, obtaining the camera coordinate system transformation matrix of the fisheye camera and the calibration camera; step 4, correcting the image to be calibrated in the overlapping area according to the camera coordinate system transformation matrix and the homography matrix. The present invention realizes the correction of the surround view system image, accurately restores the edge part of the image, and improves the product performance.

Description

Technical Field

[0001] The present invention relates to the field of vehicle-mounted surround view technology, and in particular to an image correction method for a surround view system. Background Art

[0002] At present, surround view systems are increasingly used in vehicles, greatly improving the field of vision of vehicles when parking and playing a good role in preventing scratches. However, existing vehicle-mounted surround view systems are basically based on the stitching of images taken by fisheye cameras. Since the edge distortion of fisheye images is relatively serious, the edge of the fisheye image is stretched more severely during distortion correction, resulting in a low degree of restoration of the edge of the fisheye image when stitching, affecting the use effect. Summary of the invention

[0003] The present invention provides a surround view system image correction method, which aims to solve the defects in the prior art, realize the correction of the surround view system image, accurately restore the edge part of the image, and improve product performance.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is:

[0005] The present invention provides a surround view system image correction method, comprising:

[0006] Step 1: receiving an image to be calibrated and an auxiliary calibration image, wherein the image to be calibrated is an image taken by a fisheye camera of a surround view system and subjected to distortion correction, and the auxiliary calibration image is an undistorted image taken by a calibration camera;

[0007] Step 2: aligning the image to be calibrated and the calibration auxiliary image to obtain an overlapping area;

[0008] Step 3: Obtain the camera coordinate system transformation matrix of the fisheye camera and the calibration camera;

[0009] Step 4: Correct the image to be calibrated in the overlapping area according to the camera coordinate system transformation matrix and the homography matrix.

[0010] Specifically, the preset condition is that the calibration auxiliary image taken by the calibration camera can cover the to-be-calibrated area of ​​the to-be-calibrated image taken by the fisheye cameras on both sides.

[0011] Specifically, the step 1 includes:

[0012] Step 101: Install the fisheye cameras of the surround view system at a predetermined position, and install a calibration camera between two adjacent fisheye cameras according to preset conditions;

[0013] Step 102, calibrating the fisheye camera and the calibration camera of the surround view system, and saving the calibrated intrinsic parameters, extrinsic parameters, and distortion parameters of the fisheye camera and the calibration camera;

[0014] Step 103: Control the fisheye camera and the calibration camera to capture images, and use the intrinsic parameters, extrinsic parameters, and distortion parameters to correct and transform the respective images to obtain the image to be calibrated and the calibration auxiliary image.

[0015] Specifically, the step 2 includes:

[0016] Step 201: adjusting the image resolution of the calibration auxiliary image to be consistent with the image resolution of the image to be calibrated;

[0017] Step 202: extract key feature points of the image to be calibrated and the calibration auxiliary image;

[0018] Step 203, calculating a current homography matrix according to key feature points of the image to be calibrated and the calibration auxiliary image;

[0019] Step 204: Affine transform the image to be calibrated into the calibration auxiliary image using the homography matrix to obtain an overlapping area between the two.

[0020] Specifically, the step 203 includes:

[0021] Step 2031, extracting a preset number of frames of images before the current image of the overlapping area, and arranging the matching point pairs in the preset number of frames of images in order according to the size of the Euclidean distance;

[0022] Step 2032: Calculate the similarity of each matching point pair by normalized covariance;

[0023] Step 2033, calculating the maximum value of the similarity, and taking the homography matrix of the image frame corresponding to the largest one among the maximum values ​​as the current homography matrix;

[0024] Step 2034: When the number of changes in the matching point pairs in the overlapping area exceeds a preset value, return to step 2031.

[0025] Specifically, the step 3 includes:

[0026] Step 301: Place a calibration object in the overlapping area of ​​the field of view of the calibration camera and the fisheye camera;

[0027] Step 302: Control the calibration camera and the fisheye camera to photograph the calibration object to obtain a first calibration image and a second calibration image respectively;

[0028] Step 303: establishing a first relational expression of the camera coordinate systems of the calibration camera and the fisheye camera;

[0029] Step 304: determining a second relationship according to any point on the first calibration image and its corresponding point on the second calibration image;

[0030] Step 305: Solve the camera coordinate system transformation matrix according to the first relationship and the second relationship.

[0031] Specifically, the first relational expression is:

[0032] W fish =R*W nml +T

[0033] Among them, R is the rotation matrix and T is the translation vector.

[0034] Specifically, the second relational expression is:

[0035] W nml =R nml *p+T nml

[0036] W fish =R fish *p+T fish

[0037] Among them, W nml is the spatial coordinate of any point in the calibration camera coordinate system, R nml is the rotation matrix of the calibration camera, T nml is the translation vector of the calibration camera, W fish is the spatial coordinate of any point in the calibration camera coordinate system, R fish is the rotation matrix of the calibration camera, T fish is the translation vector of the calibration camera, and p is the coordinate of any point in the world coordinate system.

[0038] Specifically, the transformation matrix is:

[0039]

[0040]

[0041] Among them, R fish represents the rotation matrix of the fisheye camera, T fish Represents the translation vector of the fisheye camera, R -1 nml represents the inverse matrix of the rotation matrix of the calibration camera, T nml Represents the translation vector of the calibrated camera.

[0042] Specifically, step 4 includes:

[0043] Step 401: setting the calibration auxiliary image as a reference image;

[0044] Step 402: determine the area to be calibrated;

[0045] Step 403, calculating a second pixel point on the calibration auxiliary image corresponding to a first pixel point of the image to be calibrated within the calibration area after being transformed by the homography matrix;

[0046] Step 404, calculating a third pixel point on the calibration auxiliary image corresponding to the first pixel point after being transformed by the transformation matrix;

[0047] Step 405: determine whether the Euclidean distance between the second pixel and the third pixel exceeds a preset distance threshold; if so, move the first pixel to the third pixel; otherwise, retain the first pixel.

[0048] The beneficial effects of the present invention are as follows: the present invention receives the image to be calibrated and the calibration auxiliary image, obtains the overlapping area between the two, and obtains the camera coordinate system transformation matrix and the homography matrix of the fisheye camera and the calibration camera, and then corrects the image to be calibrated in the overlapping area according to the camera coordinate system transformation matrix and the homography matrix, thereby realizing the correction of the surround view system image, accurately restoring the edge part of the image, and improving the product performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic flow chart of the surround view system image correction method of the present invention;

[0050] Figure 2 is a schematic diagram of the relative installation positions of the fisheye camera and the calibration camera of the present invention;

[0051] Figure 3 is a schematic diagram of imaging of a calibration camera and a fisheye camera of the present invention;

[0052] Figure 4 It is a schematic diagram of the area to be calibrated of the present invention. DETAILED DESCRIPTION

[0053] The following specifically describes the embodiments of the present invention in conjunction with the accompanying drawings, which are only for reference and illustration and do not constitute a limitation on the scope of patent protection of the present invention.

[0054] like Figure 1 As shown, this embodiment provides a surround view system image correction method, including:

[0055] Step 1: receiving an image to be calibrated and an auxiliary calibration image, wherein the image to be calibrated is an image taken by a fisheye camera of a surround view system and subjected to distortion correction, and the auxiliary calibration image is an undistorted image taken by a calibration camera.

[0056] In this embodiment, step 1 includes:

[0057] Step 101: Install fisheye cameras of a surround view system at predetermined positions, and install a calibration camera between two adjacent fisheye cameras according to preset conditions.

[0058] The preset condition is that the calibration auxiliary image taken by the calibration camera can cover the to-be-calibrated area of ​​the to-be-calibrated image taken by the fisheye cameras on both sides, that is, the area with severe distortion.

[0059] The relative installation positions of the fisheye camera and the calibration camera are as follows: Figure 2 As shown, C1-C4 are fisheye cameras, TC1-TC4 are calibration cameras, and the calibration cameras are ordinary color cameras or TOF cameras.

[0060] Step 102: calibrate the fisheye camera and the calibration camera of the surround view system, and save the calibrated intrinsic parameters, extrinsic parameters, and distortion parameters of the fisheye camera and the calibration camera.

[0061] For example, the Zhang Zhengyou calibration method can be used to calibrate the internal parameters, external parameters, and distortion parameters.

[0062] Step 103: Control the fisheye camera and the calibration camera to capture images, and use the intrinsic parameters, extrinsic parameters, and distortion parameters to correct and transform the respective images to obtain the image to be calibrated and the calibration auxiliary image.

[0063] Step 2: align the image to be calibrated and the calibration auxiliary image to obtain an overlapping area.

[0064] In this embodiment, step 2 includes:

[0065] Step 201: Adjust the image resolution of the calibration auxiliary image to be consistent with the image resolution of the image to be calibrated.

[0066] Step 202: extract key feature points of the image to be calibrated and the calibration auxiliary image.

[0067] In specific implementation, the key feature points of the image can be extracted by feature extraction algorithms such as SIFT and ORB. This is a prior art and will not be described in detail.

[0068] Step 203: Calculate the current homography matrix H according to the key feature points of the image to be calibrated and the calibration auxiliary image.

[0069] In this embodiment, step 203 includes:

[0070] Step 2031: Extract the images of the preset number of frames before the current image in the overlapping region, and arrange the matching point pairs in the preset number of frames in ascending order of the Euclidean distance;

[0071] Step 2032: Calculate the similarity of each matching point pair through normalized covariance;

[0072] Step 2033: Calculate the maximum value of the similarities, and use the homography matrix H of the image frame corresponding to the largest one among the maximum values max as the current homography matrix H;

[0073] Step 2034: When the change quantity of the matching point pairs in the overlapping region exceeds the preset value, return to Step 2031.

[0074] The above steps determine the homography matrix through the image correlation in the overlapping region, and adjust the current homography matrix in real time by means of dynamic tracking, with higher accuracy and better registration effect.

[0075] Step 204: Affinely transform the image to be calibrated into the calibration auxiliary image through the homography matrix H to obtain the overlapping region of the two.

[0076] For example, assume that the pixel matrix of the image to be calibrated is P[m][n], the pixel matrix of the calibration auxiliary image is Q[m][n], and the pixel matrix of the overlapping region obtained after Step 204 is T[i][j], where i < m, j < n, and m, n, i, j are positive integers.

[0077] Step 3: Obtain the camera coordinate system transformation matrix (R, T) of the fisheye camera and the calibration camera.

[0078] In this embodiment, Step 3 includes:

[0079] Step 301. Place a calibration object in the overlapping field of view of the calibration camera and the fisheye camera.

[0080] The calibration object is a checkerboard or other calibration objects that are easy to identify corner points and feature points.

[0081] Step 302: Control the calibration camera and the fisheye camera to photograph the calibration object to obtain a first calibration image and a second calibration image respectively.

[0082] The imaging schematic of a certain point p in the overlapping field of view of the calibration camera and the fisheye camera is as Figure 3 shown.

[0083] Step 303: Establish a first relational expression of the camera coordinate systems of the calibration camera and the fisheye camera.

[0084] W fish =R*W nml +T

[0085] Among them, R is the rotation matrix and T is the translation vector.

[0086] Step 304: Determine a second relationship according to any point on the first calibration image and its corresponding point on the second calibration image.

[0087] W nml =R nml *p+T nml

[0088] W fish =R fish *p+T fish

[0089] Among them, W nml is the spatial coordinate of any point in the calibration camera coordinate system, R nml is the rotation matrix of the calibration camera, T nml is the translation vector of the calibration camera, W fish is the spatial coordinate of any point in the calibration camera coordinate system, R fish is the rotation matrix of the calibration camera, T fish is the translation vector of the calibration camera, and p is the coordinate of any point in the world coordinate system.

[0090] Step 305: Solve the camera coordinate system transformation matrix (R, T) according to the first relationship and the second relationship.

[0091] In this embodiment, the transformation matrix (R, T) is:

[0092]

[0093]

[0094] Among them, R fish represents the rotation matrix of the fisheye camera, T fish Represents the translation vector of the fisheye camera, R -1 nml represents the inverse matrix of the rotation matrix of the calibration camera, T nml Represents the translation vector of the calibrated camera.

[0095] Step 4: Correct the image to be calibrated in the overlapping area according to the camera coordinate system transformation matrix (R, T) and the homography matrix H.

[0096] In this embodiment, step 4 includes:

[0097] Step 401: Set the calibration auxiliary image as a reference image.

[0098] Step 402: Determine the area to be calibrated.

[0099] It can be seen from the above description that the edge portion of the image to be calibrated is a severely distorted portion, that is, the edge portion of the overlapping region T[i][j] is a severely distorted region.

[0100] In the specific implementation, the area where the image to be calibrated is severely distorted is first obtained by calibration, and then the position of the area where the image to be calibrated is severely distorted within the overlapping area T[i][j] is determined, and the position is the area to be calibrated, such as Figure 4 shown.

[0101] Step 403: Calculate the first pixel point (x 1 ,y 1 ) is transformed by the homography matrix H to obtain the corresponding second pixel point (x 2 ,y 2 ).

[0102] Step 404: Calculate the first pixel point (x 1 ,y 1 ) is transformed by the transformation matrix (R, T) and corresponds to the third pixel point (x 3 ,y 3 ).

[0103] Step 405: Determine the second pixel point (x 2 ,y 2 ) and the third pixel point (x 3 ,y 3 ) exceeds the preset distance threshold, if yes, the first pixel point (x 1 ,y 1 ) moves the third pixel point (x 3 ,y 3 ), otherwise keep the first pixel (x 1 ,y 1 ).

[0104] The above steps calibrate only the image to be calibrated in the calibration area, rather than the image to be calibrated in the entire overlapping area, which is faster and can better meet the real-time requirements.

[0105] The above disclosure is only a preferred embodiment of the present invention, which cannot be used to limit the scope of protection of the present invention. Therefore, equivalent changes made according to the scope of the patent application of the present invention are still within the scope covered by the present invention.

Claims

1. A surround view system image correction method, characterized in that: include: Step 1: receiving an image to be calibrated and an auxiliary calibration image, wherein the image to be calibrated is an image taken by a fisheye camera of a surround view system and subjected to distortion correction, and the auxiliary calibration image is an undistorted image taken by a calibration camera; Step 2: registering the image to be calibrated and the calibration auxiliary image to obtain an overlapping area, including: Step 201: adjusting the image resolution of the calibration auxiliary image to be consistent with the image resolution of the image to be calibrated; Step 202: extract key feature points of the image to be calibrated and the calibration auxiliary image; Step 203, calculating a current homography matrix according to key feature points of the image to be calibrated and the calibration auxiliary image; Step 204: Affine transform the image to be calibrated into the calibration auxiliary image using the current homography matrix to obtain an overlapping area between the two. Step 3: Obtain the camera coordinate system transformation matrix of the fisheye camera and the calibration camera; Step 4: Correcting the image to be calibrated in the overlapping area according to the camera coordinate system transformation matrix and the current homography matrix; The step 4 comprises: Step 401: setting the calibration auxiliary image as a reference image; Step 402: determine the area to be calibrated; Step 403, calculating a second pixel point on the calibration auxiliary image corresponding to a first pixel point of the image to be calibrated within the calibration area after being transformed by the current homography matrix; Step 404, calculating a third pixel point on the calibration auxiliary image corresponding to the first pixel point after being transformed by the transformation matrix; Step 405 , determining whether the Euclidean distance between the second pixel and the third pixel exceeds a preset distance threshold; if so, moving the first pixel to the third pixel; otherwise, retaining the first pixel.

2. The surround view system image correction method according to claim 1, characterized in that: The step 1 comprises: Step 101: Install the fisheye cameras of the surround view system at a predetermined position, and install a calibration camera between two adjacent fisheye cameras according to preset conditions; Step 102, calibrating the fisheye camera and the calibration camera of the surround view system, and saving the calibrated intrinsic parameters, extrinsic parameters, and distortion parameters of the fisheye camera and the calibration camera; Step 103: Control the fisheye camera and the calibration camera to capture images, and use the intrinsic parameters, extrinsic parameters, and distortion parameters to correct and transform the respective images to obtain the image to be calibrated and the calibration auxiliary image.

3. The surround view system image correction method according to claim 2, characterized in that: The preset condition is that the calibration auxiliary image taken by the calibration camera can cover the to-be-calibrated area of ​​the to-be-calibrated image taken by the fisheye cameras on both sides.

4. The surround view system image correction method according to claim 1, characterized in that: The step 203 includes: Step 2031, extracting a preset number of frames of images before the current image of the overlapping area, and arranging the matching point pairs in the preset number of frames of images in order according to the size of the Euclidean distance; Step 2032: Calculate the similarity of each matching point pair by normalized covariance; Step 2033, calculating the maximum value of the similarity, and taking the homography matrix of the image frame corresponding to the largest one among the maximum values ​​as the current homography matrix; Step 2034: When the number of changes in the matching point pairs in the overlapping area exceeds a preset value, return to step 2031.

5. The surround view system image correction method according to claim 1, characterized in that: The step 3 comprises: Step 301. Placing a calibration object in the overlapping area of ​​the field of view of the calibration camera and the fisheye camera; Step 302: Control the calibration camera and the fisheye camera to photograph the calibration object to obtain a first calibration image and a second calibration image respectively; Step 303: Determine, according to the pinhole imaging model, a first relational expression of the spatial coordinates of any point on the first calibration image, the image plane projection coordinates of the calibration camera, and the calibration camera intrinsic parameter matrix; Step 303: establishing a first relational expression of the camera coordinate systems of the calibration camera and the fisheye camera; Step 304: determining a second relationship according to any point on the first calibration image and its corresponding point on the second calibration image; Step 305: Solve the camera coordinate system transformation matrix according to the first relationship and the second relationship.

6. The surround view system image correction method according to claim 5, characterized in that: The first relation is: T Among them, R is the rotation matrix, T is the translation vector, and W fish is the spatial coordinate of any point in the fisheye camera coordinate system, W nml is the spatial coordinate of any point in the calibration camera coordinate system.

7. The surround view system image correction method according to claim 6, characterized in that: The second relation is: Among them, W nml is the spatial coordinate of any point in the calibration camera coordinate system, R nml is the rotation matrix of the calibration camera, T nml is the translation vector of the calibration camera, W fish is the spatial coordinate of any point in the fisheye camera coordinate system, R fish is the rotation matrix of the calibration camera, T fish is the translation vector of the fisheye camera, and p is the coordinate of any point in the world coordinate system.

8. The surround view system image correction method according to claim 7, characterized in that: The transformation matrix is: Among them, R fish represents the rotation matrix of the fisheye camera, T fish Represents the translation vector of the fisheye camera, R -1 nml represents the inverse matrix of the rotation matrix of the calibration camera, T nml Represents the translation vector of the calibrated camera.

Images