A surround view camera calibration method and device, electronic equipment and storage medium

Abstract

The application discloses a kind of surround view camera calibration method, device, electronic equipment and storage medium, it is related to intelligent driving technical field, to improve the calibration precision, calibration efficiency and calibration success rate of surround view camera, the method comprises: first, the original image of target vehicle is obtained, and the original image is operated to obtain target image;Then, based on the geometric features of original corner point, all target corner points in target image are identified;Further, according to the pixel value of target corner point, prior world coordinate value and internal parameter, target external parameter is calculated;Finally, based on original image, target external parameter and internal parameter, the re-projection error of target image and the bird's-eye view splicing error of adjacent target camera are calculated, if re-projection error and bird's-eye view splicing error meet preset requirements, then determine that target camera calibration is completed.

Description

TECHNICAL FIELD

[0001] The present application mainly relates to the technical field of intelligent driving, and in particular relates to a surround-view camera calibration method and device, an electronic device and a storage medium. BACKGROUND

[0002] With the rapid development of intelligent driving cars, the demand for auxiliary intelligent driving functions under the visual scheme in the automobile industry has surged, making 360-degree, 540-degree and other multi-camera surround-view systems have become a regular configuration of intelligent cars, and parameter calibration of multiple cameras is the basis of the surround-view system, and the quality of parameter calibration will directly affect the final effect of the auxiliary driving system.

[0003] At present, the process of surround-view camera parameter calibration is generally divided into two parts. One is the conversion from the camera coordinate system to the image coordinate system, which is the conversion of three-dimensional points to two-dimensional points, including the intrinsic parameters of the surround-view camera. The intrinsic parameters of the surround-view camera are the description of the physical characteristics of the camera, such as focal length, resolution, etc. The second is the conversion from the world coordinate system to the camera coordinate system, which is the conversion of three-dimensional points to three-dimensional points, including the extrinsic parameters of the surround-view camera. The extrinsic parameters of the surround-view camera determine the position and orientation of the camera in a certain three-dimensional space, such as rotation parameters, translation parameters, etc.

[0004] In related technologies, when calibrating the extrinsic parameters of the surround-view camera, a checkerboard is usually used for calibration, that is, a checkerboard is set in the calibration room, the vehicle is parked at a fixed position, and the checkerboard is arranged in the front, rear, left and right directions of the vehicle, and the parameters of the checkerboard are recorded. The physical coordinates of the corner points of the checkerboard and other parameters are combined to calibrate the extrinsic parameters of the surround-view camera. However, when calibrating the extrinsic parameters of the surround-view camera by the above method, the corner points need to be selected manually, and are easily affected by the installation position of the surround-view camera and the influence of the checkerboard bearing medium (such as calibration wrinkles), resulting in inaccurate selection of corner points and the inability to select most available corner points, thereby reducing the accuracy of the calibration of the extrinsic parameters of the surround-view camera. SUMMARY

[0005] The present application provides a surround-view camera calibration method, device, electronic device and storage medium to improve the accuracy and success rate of surround-view camera calibration.

[0006] In a first aspect, the present application provides a surround-view camera calibration method, comprising:

[0007] Obtaining an original image of a target vehicle, and performing a de-distortion operation on the original image to obtain a target image;

[0008] Identifying all target corner points in the target image based on the geometric features of the original corner points; wherein the geometric features represent the linearity and symmetry relationship of the original corner points;

[0009] According to the pixel value of the target corner point, the prior world coordinate value and the intrinsic parameter, the target extrinsic parameter is calculated;

[0010] Based on the original image, the target extrinsic parameter and the intrinsic parameter, a re-projection error of the target image and a bird's eye view seam error of the adjacent target camera are calculated, and if the re-projection error and the bird's eye view seam error meet the preset requirement, it is determined that the target camera calibration is completed.

[0011] In an optional embodiment, all target corner points in the target image are identified based on the geometric features of the original corner points, including:

[0012] For the target images obtained from the front and rear directions of the target vehicle, the following operations are respectively performed:

[0013] All original corner points in the target image are obtained;

[0014] A plurality of mutually parallel horizontal straight lines are determined in the original corner points;

[0015] The original corner points symmetric about the center point on the straight line are taken as the target corner points.

[0016] In an optional embodiment, all target corner points in the target image are identified based on the geometric features of the original corner points, including:

[0017] For the target images obtained from the left and right directions of the target vehicle, the following operations are respectively performed:

[0018] All original corner points in the target image are obtained;

[0019] Edge point detection is performed on the original corner points, and all edge points in the target image are obtained;

[0020] A target rectangular region is obtained in the target image based on the edge points;

[0021] A plurality of mutually parallel horizontal straight lines are determined in the target rectangular region;

[0022] The corner points located on the straight line and the corner points of the vertices of the target rectangular region are taken as the target corner points.

[0023] In an optional embodiment, a re-projection error of the target image and a bird's eye view seam error of the adjacent target camera are calculated based on the original image, the target extrinsic parameter and the intrinsic parameter, and if the re-projection error and the bird's eye view seam error meet the preset requirement, it is determined that the target camera calibration is completed, including:

[0024] According to the target extrinsic parameter and the intrinsic parameter, inverse perspective transformation is performed on the original image to obtain a bird's eye view of the target vehicle;

[0025] A first world coordinate of the target corner point in the bird's eye view is obtained;

[0026] According to the first world coordinate, coordinate conversion is performed to obtain the first pixel coordinate of the target corner point;

[0027] The second pixel coordinate of the target corner point is obtained through a corner point detection algorithm;

[0028] The pixel value of the second pixel coordinate is subtracted from the pixel value of the first pixel coordinate to obtain a target pixel difference of the target corner point;

[0029] If the target pixel difference is less than a preset threshold, it is determined that the target camera calibration is completed.

[0030] In an optional implementation, when the re-projection error of the target image and the aerial view seam error of the adjacent target camera are calculated based on the original image, the target external parameter and the internal parameter, if the re-projection error and the aerial view seam error meet a preset requirement, it is determined that the target camera calibration is completed, and the method further comprises:

[0031] According to the target external parameter and the internal parameter, inverse perspective transformation is performed on the original image to obtain an aerial view of the target vehicle;

[0032] Fourth pixel coordinates of the common-view corner points corresponding to the common-view area in the aerial view are obtained; wherein the common-view area is an overlapping area of the acquisition ranges of two adjacent target cameras;

[0033] According to the third pixel coordinate, coordinate conversion is performed to obtain the second world coordinate of the common-view corner point;

[0034] The Euclidean distance difference between the third world coordinate of the actually measured common-view corner point and the second world coordinate is obtained;

[0035] If the Euclidean distance difference is less than a preset threshold, it is determined that the target camera calibration is completed; wherein the Euclidean distance difference represents the aerial view seam error of the adjacent target camera.

[0036] In a second aspect, the application provides a surround-view camera calibration device, comprising:

[0037] A processing module is configured to obtain an original image of a target vehicle, and perform a de-distortion operation on the original image to obtain a target image;

[0038] An identification module is configured to identify all target corner points in the target image based on the geometric features of the original corner points; wherein the geometric features represent the linearity and symmetry relationship of the original corner points;

[0039] A calculation module is configured to calculate a target external parameter according to the pixel value of the target corner point, the prior world coordinate value and the internal parameter;

[0040] The determining module is configured to calculate a re-projection error of the target image and a bird's-eye view seam error of an adjacent target camera based on the original image, the target extrinsic parameter, and the intrinsic parameter, and determine that the target camera is calibrated if the re-projection error and the bird's-eye view seam error meet preset requirements.

[0041] In an optional implementation, when all target corner points are identified in the target image based on geometric features of the original corner points, the identification module is specifically configured to:

[0042] For target images obtained from the front and rear directions of the target vehicle, the following operations are respectively performed:

[0043] All original corner points in the target image are obtained;

[0044] A plurality of horizontal straight lines parallel to each other are determined from the original corner points;

[0045] Original corner points symmetrical about the center point on the straight line are taken as target corner points.

[0046] In an optional implementation, when all target corner points are identified in the target image based on geometric features of the original corner points, the identification module is specifically configured to:

[0047] For target images obtained from the left and right directions of the target vehicle, the following operations are respectively performed:

[0048] All original corner points in the target image are obtained;

[0049] Edge point detection is performed on the original corner points, and all edge points in the target image are obtained;

[0050] A target rectangular region is obtained in the target image based on the edge points;

[0051] A plurality of horizontal straight lines parallel to each other are determined in the target rectangular region;

[0052] Corner points located on the horizontal straight lines and corner points of vertices of the target rectangular region are taken as target corner points.

[0053] In an optional implementation, when the re-projection error of the target image and the bird's-eye view seam error of the adjacent target camera are calculated based on the original image, the target extrinsic parameter, and the intrinsic parameter, and the re-projection error and the bird's-eye view seam error meet preset requirements, the determining module is specifically configured to:

[0054] According to the target extrinsic parameter and the intrinsic parameter, inverse perspective transformation is performed on the original image to obtain a bird's-eye view of the target vehicle;

[0055] First world coordinates of the target corner points on the bird's-eye view are obtained;

[0056] perform coordinate conversion according to the first world coordinates to obtain first pixel coordinates of the target corner point;

[0057] perform coordinate conversion according to the first world coordinates to obtain first pixel coordinates of the target corner point;

[0058] perform coordinate conversion according to the first world coordinates to obtain first pixel coordinates of the target corner point;

[0059] If the target pixel difference is less than a preset threshold, it is determined that the target camera calibration is completed.

[0060] In an optional embodiment, when the re-projection error of the target image and the aerial view seam error of the adjacent target camera are calculated based on the original image, the target extrinsic parameter and the intrinsic parameter, and the re-projection error and the aerial view seam error meet a preset requirement, the determining module is further configured to:

[0061] perform inverse perspective transformation on the original image according to the target extrinsic parameter and the intrinsic parameter to obtain an aerial view of the target vehicle;

[0062] obtain third pixel coordinates of a common-view corner point in the aerial view, the common-view corner point being in a common-view area of the two adjacent target cameras;

[0063] perform coordinate conversion according to the third pixel coordinates to obtain second world coordinates of the common-view corner point;

[0064] obtain a Euclidean distance difference value between the third world coordinates of the common-view corner point actually measured and the second world coordinates;

[0065] If the Euclidean distance difference value is less than a preset threshold, it is determined that the target camera calibration is completed, wherein the Euclidean distance difference value represents the aerial view seam error of the adjacent target cameras.

[0066] In a third aspect, the present application provides an electronic device, comprising:

[0067] a memory for storing a computer program;

[0068] a processor for executing the computer program stored in the memory to implement the steps of the surround-view camera calibration method.

[0069] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the surround-view camera calibration method.

[0070] By means of the technical solutions in one or more of the above embodiments of the present application, the embodiments of the present application have at least the following beneficial effects:

[0071] In the surround view camera calibration method provided in the embodiments of the present application, first, an original image of a target vehicle is acquired, and a distortion removal operation is performed on the original image to obtain a target image; the distortion removal operation restores lines and the like that are distorted, thereby improving the accuracy of subsequent image processing; then, all target corners in the target image are identified based on geometric features of original corners; the geometric features represent linearity and symmetry of the original corners, and the selection of the target corners based on the geometric features of the original corners avoids the problems of inaccurate selection of target corners and fewer available target corners caused by manual selection of corners and selection of corners based on connectivity in related technologies; further, target extrinsic parameters are calculated based on the target corners and intrinsic parameters; finally, a re-projection error of the target image is calculated based on the original image, the target extrinsic parameters and the intrinsic parameters, and if the re-projection error meets preset requirements, it is determined that the target camera calibration is completed.

[0072] In this way, the target corners are selected based on the geometric features of the original corners, which can avoid the problem that connectivity cannot identify corners. When the target corners are selected, the target corners are automatically selected based on the geometric features of the original corners, and the target corners are uniformly and symmetrically distributed in space, which improves the accuracy of the surround view camera calibration, completely avoids manual intervention, avoids low efficiency caused by incomplete automation of manual selection of corners, and avoids the problem that most available corners cannot be selected due to wrinkles in calibration based on connectivity, which causes unsuccessful calibration. When the target corners are selected, corners with low identification accuracy can be automatically removed, which improves the accuracy of calibration. While improving the accuracy of the surround view camera calibration, the re-projection error of the target image is calculated, the coordinates are converted between the world coordinate system and the pixel coordinate system, and the difference between the converted coordinates and the actually detected coordinates is quantified, which fully quantifies the accuracy of the surround view camera calibration.

[0073] The above-mentioned aspects and the technical effects that can be achieved by the aspects will be described in detail with reference to the technical effects that can be achieved by the first aspect and the possible solutions of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0074] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. In the drawings:

[0075] Figure 1 An implementation flowchart of a surround view camera calibration method provided in the embodiments of the present application;

[0076] Figure 2 A schematic diagram of determining a horizontal straight line provided for an embodiment of the present application;

[0077] Figure 3 A schematic diagram of a common view area of a surround view camera provided for an embodiment of the present application;

[0078] Figure 4 A schematic diagram of a Delaunay triangulation network of a target corner point provided for an embodiment of the present application;

[0079] Figure 5 A schematic diagram of a missed detection of a target corner point provided for an embodiment of the present application;

[0080] Figure 6 A schematic diagram of a false detection of a target corner point provided for an embodiment of the present application;

[0081] Figure 7 A schematic diagram of a Delaunay triangulation network of a target corner point under another target pattern provided for an embodiment of the present application;

[0082] Figure 8 A schematic diagram of a surround view camera calibration device provided for an embodiment of the present application;

[0083] Figure 9 A schematic diagram of an electronic device structure provided for an embodiment of the present application. DETAILED DESCRIPTION

[0084] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments described in the present application document, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application technical solutions.

[0085] It should be noted that in the description of the present application, “multiple” is understood as “at least two”. The association relationship of “and / or” describing the associated objects means that there can be three relationships, for example, A and / or B can represent the three cases of A existing alone, A and B existing together, and B existing alone. A is connected with B, which can represent two cases: A is directly connected with B and A is connected with B through C. In addition, in the description of the present application, “first”, “second”, etc. are used only for the purpose of distinguishing the description, and cannot be understood as indicating or implying relative importance, nor can it be understood as indicating or implying order.

[0086] The embodiments of the present application will be described in detail below with reference to the drawings.

[0087] With the rapid development of intelligent driving cars, the demand for the function of assisting intelligent driving under the visual solution in the automobile industry has increased rapidly, so that the surround view system of multi-cameras such as 360 degrees and 540 degrees has become a regular configuration of intelligent cars, and the parameter calibration of multi-cameras is the basis of the surround view system, and the quality of the parameter calibration will directly affect the final effect of the assisted driving system.

[0088] At present, the process of surround camera parameter calibration is generally divided into two parts. One is the conversion from the camera coordinate system to the image coordinate system, which is the conversion from three-dimensional points to two-dimensional points, including the camera internal parameters. Among them, the surround camera internal parameters are the description of the physical characteristics of the camera, such as focal length, resolution, etc. The second is the conversion from the world coordinate system to the camera coordinate system, which is the conversion from three-dimensional points to three-dimensional points, including the external parameters of the surround camera. Among them, the external parameters of the surround camera determine the position and orientation of the camera in a certain three-dimensional space, such as rotation parameters, translation parameters, etc.

[0089] In the related art, when calibrating the external parameters of the surround camera, a checkerboard is usually used for calibration, that is, a checkerboard is set in the calibration room, the vehicle is parked at a fixed position, and the checkerboard is arranged in the front, rear, left and right directions of the vehicle, and the physical coordinates and other parameters of the corner points of the checkerboard are recorded to calibrate the external parameters of the surround camera. However, when calibrating the external parameters of the surround camera by the above method, the corner points need to be selected manually, and are easily affected by the installation position of the surround camera and the influence of the checkerboard bearing medium (such as calibration wrinkles), resulting in inaccurate selection of corner points and most available corner points cannot be selected, thereby reducing the accuracy of the calibration of the external parameters of the surround camera.

[0090] Therefore, in order to solve the above technical problems, the present application provides a surround camera calibration method, which comprises the following steps: first, obtaining an original image of a target vehicle, and performing a de-distortion operation on the original image to obtain a target image; then, based on the geometric characteristics of the original corner points, all target corner points in the target image are identified; further, the target external parameters are calculated according to the target corner points and the internal parameters; finally, the re-projection error of the target image is calculated based on the original image, the target external parameters and the internal parameters, and if the re-projection error meets the preset requirements, it is determined that the target camera calibration is completed. Through the above method, the available corner points can be effectively selected, and manual selection is not required, thereby avoiding the problem of inaccurate selection of corner points and the problem of too few available corner points caused by selecting corner points based on connectivity.

[0091] It should be noted that the preferred embodiments of the present application are described below in conjunction with the accompanying drawings, and the preferred embodiments described herein are only used to explain and illustrate the present application, and are not used to limit the present application, and the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0092] Referring to Figure 1 An implementation flowchart of a surround view camera calibration method provided by an embodiment of the present application is shown in the figure:

[0093] S1: Obtain an original image of a target vehicle, and perform a de-distortion operation on the original image to obtain a target image.

[0094] In an embodiment of the present application, when calibrating the external parameters of each surround view camera in the surround view system of the target vehicle, a checkerboard (or other target pattern, such as a circular hole) can be arranged around the target vehicle.

[0095] And, the surround view cameras are installed on the body of the target vehicle, for example, surround view cameras are installed in the front, rear, left and right directions of the body of the target vehicle, and the original image of the target vehicle containing the checkerboard can be collected by each surround view camera. It can be understood that when collecting the original image, it is collected from the front, rear, left and right directions of the target vehicle respectively, so that the original image of the target vehicle in the front, rear, left and right directions can be collected.

[0096] Due to the manufacturing precision and assembly process of the surround view camera, or due to the angle, rotation, scaling, etc. when shooting the original image, the collected original image often has a certain distortion. Therefore, the original image collected by each surround view camera needs to be de-distorted. The distorted lines in each original image are restored to obtain a target image.

[0097] By the above method, the de-distortion operation is performed on the original image, the perspective correction of the original image is realized, and the recognition accuracy of the corner points is improved, which can improve the calibration accuracy of the external parameters of the surround view camera.

[0098] S2: Identify all target corner points in the target image based on the geometric features of the original corner points.

[0099] In an embodiment of the present application, the geometric features represent the linearity and symmetry relationship of the original corner points. Since the viewing angles of the surround view cameras in the front, rear, left and right directions are different, different methods are used to obtain the target corner points in the original image for the surround view cameras in different directions.

[0100] In an alternative implementation, for the target images obtained from the front and rear directions of the target vehicle, the following operations are performed respectively:

[0101] Firstly, the original image is converted into a gray image, and then a region of interest (ROI) is demarcated on the gray image and a mask is set. Further, an original corner point and a pixel value of a coordinate of the original corner point are detected using a corner detection operator (for example, a Tomas operator, a Sobel operator) in the ROI region. In order to improve the corner point recognition accuracy, sub-pixel corner point detection is performed by setting a Gaussian window size, a residual convergence minimum value and a maximum iteration number, and the pixel coordinate of the original corner point is sub-pixel purified.

[0102] Next, a random sample consensus (RANSAC) algorithm is used to determine a plurality of mutually parallel horizontal straight lines in the processed original corner points. As shown in FIG. 2, two original corner points are first randomly found, connected as a straight line, as a hypothetical fitting straight line L, and then points within a certain distance threshold around the straight line L are classified into the straight line, thereby obtaining a complete straight line. When the horizontal straight line is determined, the inclination of the straight line also needs to be judged. If the inclination of the straight line is greater than a certain threshold, the straight line does not meet the requirements. Through the above method, a plurality of mutually parallel horizontal straight lines can be determined in the original corner points. Figure 2

[0103] Further, a center point is determined on each of the plurality of horizontal straight lines, and original corner points symmetrical about the respective center points on the plurality of horizontal straight lines are taken as target corner points.

[0104] In the embodiment of the application, when the center point is selected, the intersection point of the horizontal straight line and the ground vertical line of the ring camera optical axis is first determined, and then the point closest to the intersection point is determined as the center point.

[0105] In an alternative embodiment, for the target images obtained from the left and right directions of the target vehicle, the following operations are respectively performed:

[0106] Firstly, all original corner points in the target image are obtained. Specifically, in order to enhance the image information and facilitate clearer expression of texture details, edges and other information in the target image, the target image is first converted into a gray image. Then, a ROI region is demarcated on the gray image and a mask is set, so as to reduce the processing time of the gray image and improve the processing accuracy. Further, image gradient calculation and gradient threshold judgment are performed on each pixel point in the ROI region on the gray image, so as to detect the edge points in the ROI region on the gray image.

[0107] ​In addition, the ROI region of the grayscale image is gridded (Grid Transform). Then, based on the edge points in the grid, straight lines are fitted to obtain the intersection points of the pairs of intersecting straight lines. By performing gradient calculations on the straight lines, the target rectangular region is obtained in the target image based on the intersection points of the pairs of intersecting straight lines.

[0108] In this embodiment of the application, after obtaining the target rectangular region in the target image, the RANSAC algorithm is also used to determine multiple parallel horizontal lines in the target rectangular region.

[0109] After determining the target rectangular area and multiple horizontal lines, the four vertices of the target rectangular area and the original corner points located on the multiple horizontal lines are taken as the target corner points.

[0110] S3: Calculate the target extrinsic parameters based on the pixel values ​​of the target corner points, the prior world coordinates, and the intrinsic parameters.

[0111] In one alternative implementation, see [link to relevant documentation] Figure 3 As shown, surround-view cameras are installed in the front, rear, left, and right directions of the target vehicle. A is the front surround-view camera, B is the rear surround-view camera, C is the left surround-view camera, and D is the right surround-view camera. In this embodiment, the surround-view cameras can be 180-degree, 360-degree, or other angled fisheye cameras. Each of the four surround-view cameras (A, B, C, and D) can acquire a field of view. There is a shared field of view between the fields of view of any two adjacent surround-view cameras. For example, there is a shared field of view between surround-view cameras A and C, or between surround-view cameras A and D.

[0112] Furthermore, the pixel values ​​and prior world coordinates corresponding to the target corner points are determined. In this embodiment, when calculating the target extrinsic parameters, the target corner points include common viewing angle points in the common viewing area and single-camera corner points in the respective viewing areas of the target cameras. Then, based on the pixel values, prior world coordinates, and the intrinsic parameters of the surround-view camera provided by the surround-view camera manufacturer, the target extrinsic parameters are calculated.

[0113] In this embodiment, the calibration of the surround-view camera is mainly achieved by converting the target corner points between the world coordinate system and the pixel coordinate system to calculate the pose.

[0114] Specifically, the transformation relationship between the world coordinate system and the pixel coordinate system is as follows:

[0115]

[0116] in, This indicates the internal parameters of the surround-view camera. The ring camera extrinsic parameter is represented.

[0117] In the embodiment of the present application, after obtaining the pixel coordinates of the target corner point, the coordinates of the target corner point in the world coordinate system, and the pixel coordinates of the co-view corner point in the image plane and the intrinsic parameter are known, the target extrinsic parameter of the ring camera can be solved by PNP (Perspective-n-Points, PNP).

[0118] In the embodiment of the present application, after the target extrinsic parameter of the ring camera is solved by PNP, an optimizer target extrinsic parameter can be constructed for optimization.

[0119] Specifically, the target extrinsic parameter obtained includes three rotational degrees of freedom and three translational degrees of freedom. By constructing a loss function, the six degrees of freedom are iteratively converged. The 3D points are converted to the camera coordinate system, normalized and converted to the image coordinate system. The error between the projection points and the image points is calculated. The optimizer parameters and the optimization method (such as the LM algorithm) are set to call the solver for optimization. The obtained optimization parameters are used as the final rotational degrees of freedom and translational degrees of freedom.

[0120] S4: Based on the original image, the target extrinsic parameter and the intrinsic parameter, the re-projection error of the target image and the aerial view seam error of the adjacent target camera are calculated. If the re-projection error and the aerial view seam error meet the preset requirements, it is determined that the target camera calibration is completed.

[0121] Since the ring camera has been calibrated for the intrinsic parameter before being installed on the target vehicle, after obtaining the target extrinsic parameter of the ring camera, inverse perspective stitching operation can be performed according to the original image, the target extrinsic parameter and the intrinsic parameter to obtain the Bird’s Eye View (BEV) of the target vehicle.

[0122] Then, the first world coordinates of the target corner point in the aerial view are obtained, and the first world coordinates of the target corner point are converted to the first pixel coordinates by coordinate conversion. In the embodiment of the present application, the second pixel coordinates (actually measured pixel coordinates) of the target corner point can also be obtained by using a corner point detection algorithm. After obtaining the first pixel coordinates and the second pixel coordinates of the target corner point, the pixel value of the second pixel coordinates is subtracted from the pixel value of the first pixel coordinates to obtain the target pixel difference of the target corner point. Further, it is judged whether the target pixel difference is less than a preset threshold value, for example, the target pixel difference is 1 pixel and the preset threshold value is 2 pixels. If the target pixel difference is less than the preset threshold value, it is determined that the target camera calibration is completed. It should be noted that the target corner point used when determining the target pixel difference of the target corner point includes the co-view corner point in the co-view region of the target camera and the single-camera corner point in the corresponding field of view range of the target camera.

[0123] In an optional embodiment, after obtaining the bird's eye view of the target vehicle, third pixel coordinates of the common view angle point in the bird's eye view are acquired; the third pixel coordinates of the common view angle point are converted into second world coordinates of the common view angle point through coordinate conversion; the Euclidean distance difference between the third world coordinates and the second world coordinates of the common view angle point is calculated, and if the Euclidean distance difference between the third world coordinates and the second world coordinates is less than a preset threshold, that is, the bird's eye view splicing error of the adjacent target camera is less than the preset threshold, it is determined that the target camera calibration is completed. It should be noted that when determining the bird's eye view splicing error of the adjacent target camera, the target angle point used only includes the common view angle point between the adjacent target cameras.

[0124] The mapping between the target angle point and 3D-2D is characterized, and the accuracy of the surround view camera calibration is quantified from the pixel difference and the Euclidean distance difference.

[0125] Similarly, after the target camera calibration is completed, the other three surround view cameras on the target vehicle are also calibrated in the same way.

[0126] In the embodiments of the present application, after the target angle point is determined based on the geometric characteristics of the original angle point, the extrinsic parameters of the surround view camera can be further calculated, and the accuracy of the surround view camera calibration is quantified by calculating the pixel difference and the Euclidean distance difference of the target angle point.

[0127] Further, in the embodiments of the present application, it can also be judged whether the target angle point is missed or misdetected. Referring to Figure 4 shown, after the target angle point is determined, a Delaunay triangulation network is constructed on the target image, and then the two-dimensional pixel point coordinate values of the final target angle point are found in the Delaunay triangulation network according to the number, row number and column number of the prior known angle points.

[0128] By constructing a Delaunay triangulation network on the target image, the parameter information of the preset target pattern can be obtained, such as a plurality of rectangular blocks in the Delaunay triangulation network, and the number, row number and column number of each rectangular block. Referring to Figure 5 shown, if a rectangular block in a certain row and column is missing, for example, Figure 5 the rectangular block in the first row and the first column is missing, it indicates that the detection of the target angle point this time has missed.

[0129] Referring to Figure 6 shown, if a rectangular block appears in the area outside the Delaunay triangulation network, that is, after the Delaunay triangulation network is determined in the target image by the number, row number and column number of the prior known angle points and the target angle point, such as Figure 6As shown, the presence of other rectangular blocks outside the Delaunay triangulation network indicates a false detection of the target corner point. Figure 6 As shown, when selecting target corner points based on the linear relationship of the original corner points, corner point 1 is not a corner point symmetrical about the symmetrical point on the line, and should not appear in the Delaunay triangulation network. Therefore, Figure 6 Corner point 1 in the image is a falsely detected corner point. This can be determined using the Delaunay triangulation network and prior pixel differences. Figure 6 Corner point number 2 in the image is also a falsely detected corner point, for example, Figure 6 The pixel differences between corner points 5 and 4, and between corner points 6 and 3 are equal, while the pixel differences between corner points 4 and 2, or between corner points 3 and 2, are too small. Therefore, it can be determined that corner point 2 is also a falsely detected corner point.

[0130] In addition, when extracting target corner points based on geometric features, the target pattern set around the target vehicle is not limited to a checkerboard pattern, but can also be other set shapes, such as circular holes.

[0131] Therefore, if the target pattern is a circular hole, when extracting the target corner point, refer to... Figure 7 As shown, a Delaunay triangulation network can also be constructed on the target image. Similarly, if the target pattern is a circular hole, the leakage or false detection of the target corner points can be determined in the same way as described above.

[0132] The above methods avoid the problems of inaccurate manual corner selection leading to low accuracy in surround-view camera calibration, and the issue of selecting corners based on connectivity failing to select many available corners. Improving the success rate of surround-view camera calibration by identifying missed and false detections of target corners enhances the accuracy of the calibration process.

[0133] Based on the same inventive concept, this application also provides a surround-view camera calibration device, see reference. Figure 8 As shown, the device includes: a processing module 801, an identification module 802, a calculation module 803, and a determination module 804; wherein,

[0134] The processing module 801 is used to acquire the original image of the target vehicle and perform distortion correction on the original image to obtain the target image.

[0135] The recognition module 802 is used to identify all target corner points in the target image based on the geometric features of the original corner points; wherein, the geometric features represent the linear and symmetrical relationships of the original corner points;

[0136] The calculation module 803 is used to calculate the target extrinsic parameters based on the pixel values ​​of the target corner points, the prior world coordinate values, and the intrinsic parameters.

[0137] determining module, configured to calculate a re-projection error of the target image and a bird's eye view seam error of an adjacent target camera based on the original image, the target extrinsic parameter and the intrinsic parameter, and determine that the target camera is calibrated if the re-projection error and the bird's eye view seam error meet preset requirements.

[0138] In an optional implementation, when all target corner points are identified in the target image based on geometric features of the original corner points, the identification module 802 is specifically configured to:

[0139] For target images obtained from the front and rear directions of the target vehicle, the following operations are respectively performed:

[0140] obtain all original corner points in the target image;

[0141] determine a plurality of horizontal straight lines parallel to each other in the original corner points;

[0142] take the original corner points symmetrical about the center point on the straight line as target corner points.

[0143] In an optional implementation, when all target corner points are identified in the target image based on geometric features of the original corner points, the identification module 802 is specifically configured to:

[0144] For target images obtained from the left and right directions of the target vehicle, the following operations are respectively performed:

[0145] obtain all original corner points in the target image;

[0146] perform edge point detection on the original corner points and obtain all edge points in the target image;

[0147] obtain a target rectangular region in the target image based on the edge points;

[0148] determine a plurality of horizontal straight lines parallel to each other in the target rectangular region;

[0149] take the corner points located on the straight line and the corner points of the vertices of the target rectangular region as target corner points.

[0150] In an optional implementation, when the re-projection error of the target image and the bird's eye view seam error of the adjacent target camera are calculated based on the original image, the target extrinsic parameter and the intrinsic parameter, and the re-projection error and the bird's eye view seam error meet preset requirements, the determination module 404 is specifically configured to:

[0151] perform inverse perspective transformation on the original image according to the target extrinsic parameter and the intrinsic parameter to obtain a bird's eye view of the target vehicle;

[0152] obtain a first world coordinate of the target corner point on the bird's eye view;

[0153] perform coordinate conversion according to the first world coordinate to obtain a first pixel coordinate of the target corner point;

[0154] obtain a second pixel coordinate of the target corner point through a corner point detection algorithm;

[0155] obtain a pixel difference of the target corner point by subtracting a pixel value of the second pixel coordinate from a pixel value of the first pixel coordinate;

[0156] If the pixel difference is less than a preset threshold, it is determined that the target camera calibration is completed.

[0157] In an optional implementation, when the re-projection error of the target image is calculated based on the original image, the target extrinsic parameter and the intrinsic parameter, and the re-projection error meets a preset requirement, the determining module 804 is further configured to:

[0158] perform inverse perspective transformation on the original image according to the target extrinsic parameter and the intrinsic parameter to obtain a bird's eye view of the target vehicle;

[0159] obtain a third pixel coordinate of the target corner point in the bird's eye view;

[0160] perform coordinate conversion according to the third pixel coordinate to obtain a second world coordinate of the target corner point;

[0161] obtain a Euclidean distance difference value between the third world coordinate and the second world coordinate of the actually measured target corner point;

[0162] If the Euclidean distance difference value is less than a preset threshold, it is determined that the target camera calibration is completed.

[0163] It should be noted that the above apparatus provided by the embodiments of the present application can realize all the method steps in the above-mentioned surround-view camera calibration method embodiments and achieve the same technical effects, and thus the same parts and beneficial effects as the method embodiments are not described in detail herein.

[0164] Based on the same inventive concept, the embodiments of the present application further provide an electronic device, which can realize the functions of the above-mentioned surround-view camera calibration method, and refer to Figure 9 The electronic device includes:

[0165] at least one processor 901 and a memory 902 connected with the at least one processor 901, and the specific connection medium between the processor 901 and the memory 902 is not limited in the embodiments of the present application, Figure 9 for example, the processor 901 and the memory 902 are connected through a bus 900. The bus 900 is connected with Figure 9The connection between the other components is indicated by a thick line, which is only illustrative and not limited. The bus 900 can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 9 The bus is indicated by a thick line, but it does not mean that there is only one bus or only one type of bus. Alternatively, the processor 901 can also be called a controller, and the name is not limited.

[0166] In the embodiment of the application, the memory 902 stores instructions executable by the at least one processor 901, and the at least one processor 901 can execute the ring camera calibration method discussed above by executing the instructions stored in the memory 902. The processor 901 can implement the functions of each module in the device shown in the figure. Figure 8

[0167] The processor 901 is the control center of the device, and can connect each part of the control device through various interfaces and lines. By running or executing the instructions stored in the memory 902 and calling the data stored in the memory 902, the device can process data and perform various functions, thereby monitoring the device as a whole.

[0168] In a possible design, the processor 901 can include one or more processing units, and the processor 901 can integrate an application processor and a modem processor. The application processor mainly processes the operating system, user interface, and application program, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 901. In some embodiments, the processor 901 and the memory 902 can be implemented on the same chip, and in some embodiments, they can also be implemented on separate chips respectively.

[0169] The processor 901 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, which can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the ring camera calibration method disclosed in the embodiments of the application can be directly embodied by a hardware processor for execution, or executed by a combination of hardware and software modules in the processor.

[0170] ​The memory 902, as a non-volatile computer readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 902 can include at least one type of storage medium, for example, can include flash memory, hard disk, multimedia card, card type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. The memory 902 is any other medium capable of carrying or storing desired program codes in the form of instructions or data structures and capable of being accessed by a computer, but is not limited thereto. The memory 902 in the embodiments of the present application can also be a circuit or any other device capable of realizing a storage function, used for storing program instructions and / or data.

[0171] By designing and programming the processor 901, the codes corresponding to the surround camera calibration method introduced in the foregoing embodiments can be fixed into the chip, so that the chip can execute the steps of the surround camera calibration method of the embodiments shown in the running time. Figure 1 How to design and program the processor 901 is a technology known to those skilled in the art, which will not be described here.

[0172] Based on the same inventive concept, the embodiments of the present application also provide a storage medium storing computer instructions, when the computer instructions run on a computer, the computer instructions make the computer execute the surround camera calibration method discussed above.

[0173] In some possible implementations, various aspects of the surround camera calibration method provided by the present application can also be implemented in the form of a program product, which includes program codes, when the program product runs on a device, the program codes are used to make the control device execute the steps in the surround camera calibration method according to various exemplary embodiments of the present application described above in the specification.

[0174] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0175] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0176] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0177] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0178] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the apparent to those skilled in the art that various modifications and changes can be made thereto without departing from the scope and spirit of the present application. It is intended that the scope of the present application be limited only by the appended claims, alongside with the full scope of equivalents to each claim.

Claims

1. A surround view camera calibration method, characterized in that, The method comprises: obtaining an original image of a target vehicle, and performing a de-distortion operation on the original image to obtain a target image; identifying all target corner points in the target image based on geometric features of original corner points; wherein the geometric features represent linearity and symmetry of the original corner points; calculating a target external parameter according to pixel values of the target corner points, prior world coordinate values, and internal parameters; calculating a re-projection error of the target image and a aerial view seam error of an adjacent target camera based on the original image, the target external parameter, and the internal parameters, and determining that target camera calibration is completed if the re-projection error and the aerial view seam error meet preset requirements, comprising: performing inverse perspective transformation on the original image according to the target external parameter and the internal parameters to obtain an aerial view of the target vehicle; obtaining first world coordinates of target corner points in the aerial view; obtaining first pixel coordinates of the target corner points through coordinate conversion based on the first world coordinates; obtaining second pixel coordinates of the target corner points through a corner point detection algorithm; obtaining a target pixel difference of the target corner points by subtracting pixel values of the first pixel coordinates from pixel values of the second pixel coordinates; determining that target camera calibration is completed if the target pixel difference is less than a preset threshold.

2. The method of claim 1, wherein, The method comprises: for target images obtained from front and rear directions of the target vehicle, the following operations are performed respectively: obtaining all original corner points in the target images; determining a plurality of horizontal straight lines parallel to each other in the original corner points; regarding original corner points symmetric about a center point on the straight lines as target corner points.

3. The method of claim 1, wherein, The method comprises: for target images obtained from left and right directions of the target vehicle, the following operations are performed respectively: obtaining all original corner points in the target images; performing edge point detection on the original corner points to obtain all edge points in the target images; obtaining a target rectangular region in the target images based on the edge points; determining a plurality of horizontal straight lines parallel to each other in the target rectangular region; regarding corner points located on the straight lines and corner points of vertices of the target rectangular region as target corner points.

4. The method of claim 1, wherein, In the method, the re-projection error of the target image and the aerial view seam error of the adjacent target camera are calculated based on the original image, the target external parameter, and the internal parameters, and it is determined that target camera calibration is completed if the re-projection error and the aerial view seam error meet preset requirements, the method further comprises: performing inverse perspective transformation on the original image according to the target external parameter and the internal parameters to obtain an aerial view of the target vehicle; obtaining third pixel coordinates of a common-view corner point corresponding to a common-view region in the aerial view; wherein the common-view region is an overlapping region of collection ranges of two adjacent target cameras; obtaining second world coordinates of the common-view corner point through coordinate conversion based on the third pixel coordinates; acquiring a third world coordinate of the actual measured common-view corner point, and a Euclidean distance difference value of the second world coordinate; if the Euclidean distance difference value is less than a preset threshold, determining that the target camera calibration is completed; wherein the Euclidean distance difference value represents a bird's-eye view seam error of the adjacent target camera.

5. A surround view camera calibration apparatus, comprising: The device comprises: a processing module configured to acquire an original image of a target vehicle, and perform a de-distortion operation on the original image to obtain a target image; an identification module configured to identify all target corner points in the target image based on geometric features of original corner points; wherein the geometric features represent linearity and symmetry of the original corner points; a calculation module configured to calculate target extrinsic parameters based on pixel values of the target corner points, prior world coordinate values, and intrinsic parameters; a determination module configured to calculate a re-projection error of the target image and a bird's-eye view seam error of an adjacent target camera based on the original image, the target extrinsic parameters, and the intrinsic parameters, and determine that the target camera calibration is completed if the re-projection error and the bird's-eye view seam error meet preset requirements, including: performing inverse perspective transformation on the original image based on the target extrinsic parameters and the intrinsic parameters to obtain a bird's-eye view of the target vehicle; acquiring first world coordinates of the target corner points on the bird's-eye view; performing coordinate conversion based on the first world coordinates to obtain first pixel coordinates of the target corner points; obtaining second pixel coordinates of the target corner points through a corner point detection algorithm; obtaining a target pixel difference of the target corner points by subtracting pixel values of the second pixel coordinates from pixel values of the first pixel coordinates; if the target pixel difference is less than a preset threshold, determining that the target camera calibration is completed.

6. The apparatus of claim 5, wherein, In the identification of all target corner points in the target image based on geometric features of original corner points, the identification module is specifically configured to: for target images acquired from front and rear directions of the target vehicle, respectively, perform the following operations: acquire all original corner points in the target images; determine multiple mutually parallel horizontal straight lines in the original corner points; take original corner points symmetric about a center point on the straight lines as target corner points.

7. The apparatus of claim 5, wherein, In the identification of all target corner points in the target image based on geometric features of original corner points, the identification module is specifically configured to: for target images acquired from left and right directions of the target vehicle, respectively, perform the following operations: acquire all original corner points in the target images; perform edge point detection on the original corner points, and obtain all edge points in the target images; obtain a target rectangular region in the target images based on the edge points; determine multiple mutually parallel horizontal straight lines in the target rectangular region; take corner points located on the straight lines and corner points of vertices of the target rectangular region as target corner points.

8. The apparatus of claim 5, wherein, In the calculation of a re-projection error of the target image and a bird's-eye view seam error of an adjacent target camera based on the original image, the target extrinsic parameters, and the intrinsic parameters, and the determination that the target camera calibration is completed if the re-projection error and the bird's-eye view seam error meet preset requirements, the determination module is further configured to: According to the target extrinsic parameter and the intrinsic parameter, inverse perspective transformation is performed on the original image to obtain an aerial view of the target vehicle; Third pixel coordinates of a common-view angle point corresponding to a common-view area in the aerial view are obtained, wherein the common-view area is an overlapping area of the acquisition ranges of two adjacent target cameras Second world coordinates of the common-view angle point are obtained through coordinate conversion according to the third pixel coordinates; A third world coordinate of the common-view angle point actually measured is obtained, and a Euclidean distance difference value of the second world coordinate is obtained; If the Euclidean distance difference value is less than a preset threshold value, it is determined that the target camera calibration is completed, wherein the Euclidean distance difference value represents an aerial view seam error of the adjacent target cameras.

9. An electronic device, comprising: Comprise: A memory for storing a computer program; A processor for executing the computer program stored on the memory to implement the method steps of any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer program stored in the computer readable storage medium is executed by the processor to implement the method steps of any one of claims 1-4.

Images