Camera calibration method, device, image stitching method, camera and vehicle

Abstract

The present disclosure relates to a camera calibration method, device, image stitching method, camera, and vehicle. The calibration method includes: obtaining a first image captured by a camera to be calibrated; wherein the first image includes a reference calibration pattern; determining the external parameters of the camera to be calibrated based on the first image; back-projecting reference points in the reference calibration pattern based on the external parameters to determine a back-projection error; when it is determined based on the back-projection error that an adjustment condition is satisfied, adjusting the target parameters of the reference calibration pattern, and returning to obtain a second image captured by the camera to be calibrated, including the reference calibration pattern after the target parameters are adjusted, until the back-projection error determined based on the second image does not satisfy the adjustment condition, at which point the calibration is completed. Thus, the method overcomes the influence of objective factors such as the reference calibration pattern on camera calibration, improves the calibration accuracy of the camera's external parameters, and helps reduce the stitching error of image stitching, thereby helping to improve the image stitching effect.

Description

Technical Field

[0001] The present disclosure relates to the technical field of camera calibration, and in particular to a camera calibration method, device, image stitching method, camera, and vehicle. Background Art

[0002] With the development of vehicle electronic technology, people have an increasing demand for assisted driving functions. Among them, the vehicle surround view function is a basic function in vehicle assisted driving and can be realized using the vehicle's panoramic surround view system.

[0003] In-vehicle panoramic surround view systems, specific scene calibration patterns are typically used to provide reference points for calibrating the extrinsic parameters (i.e., external parameters) of the system's cameras. This allows for registration of video images captured by four to eight cameras (i.e., cameras) mounted around the vehicle body, achieving panoramic surround view stitching. The accuracy of extrinsic calibration is one of the primary factors affecting image stitching errors. Typically, low extrinsic calibration accuracy results in large image stitching errors and poor image stitching results. Summary of the Invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a camera calibration method, device, image stitching method, camera and vehicle.

[0005] The present disclosure provides a camera calibration method, which includes:

[0006] Acquire a first image captured by the camera to be calibrated; wherein the first image includes a reference calibration pattern;

[0007] Determining external parameters of the camera to be calibrated based on the first image;

[0008] Back-projecting reference points in the reference calibration pattern based on the extrinsic parameters to determine a back-projection error;

[0009] When it is determined based on the back projection error that the adjustment condition is met, the target parameters of the reference calibration pattern are adjusted, and the second image of the reference calibration pattern including the adjusted target parameters, acquired by the camera to be calibrated, is returned to obtain, until the back projection error determined based on the second image no longer meets the adjustment condition, at which point the calibration is completed.

[0010] In some embodiments, the camera to be calibrated has known intrinsic parameters and distortion parameters;

[0011] The step of determining the external parameters of the camera to be calibrated based on the first image includes:

[0012] determining coordinates of reference points in the reference calibration pattern based on the first image;

[0013] Obtaining the actual three-dimensional coordinates corresponding to the reference point coordinates;

[0014] Determining the extrinsic parameters of the camera to be calibrated based on the reference point coordinates, the actual three-dimensional coordinates, the intrinsic parameters, and the distortion parameters;

[0015] The back-projecting of the reference points in the reference calibration pattern based on the external parameters to determine the back-projection error includes:

[0016] Back-projecting to obtain calculated coordinates corresponding to the reference point coordinates based on the actual three-dimensional coordinates, the internal parameters, the distortion parameters, and the external parameters;

[0017] A back-projection error is determined based on the reference point coordinates and the corresponding calculated coordinates.

[0018] In some embodiments, the number of the reference point is at least one, the coordinates of the reference point are (x, y), and the calculated coordinates are (X, Y);

[0019] The step of determining the back-projection error based on the reference point coordinates and the corresponding calculated coordinates includes:

[0020] The back-projection error is calculated using the following formula:

[0021]

[0022] Wherein, L represents the back-projection error, n represents the number of reference points, i represents the i-th reference point, n ≥ 1 and n is an integer, and i takes an integer value from 1 to n.

[0023] In some embodiments, the number of the cameras to be calibrated is at least one, and the number of the reference calibration patterns is at least one;

[0024] When the number of the reference calibration patterns collected by each camera to be calibrated is j, determining the back projection error based on the reference point coordinates and the corresponding calculated coordinates includes:

[0025] determining a back-projection error based on the coordinates of the reference points in the m reference calibration patterns and the corresponding calculated coordinates;

[0026] Wherein, j≥1 and j is an integer, and m takes integer values from 1 to j in sequence.

[0027] In some embodiments, the reference calibration pattern comprises a checkerboard, and the reference point in the reference calibration pattern comprises at least one corner point of the checkerboard;

[0028] The reference calibration pattern is set on a calibration object; correspondingly, the calibration object includes a checkerboard cloth with checkerboard patterns or a ground with checkerboard patterns;

[0029] Wherein, adjusting the target parameters of the reference calibration pattern includes:

[0030] The calibration object is adjusted.

[0031] In some embodiments, the calibration object includes M checkerboards surrounding the target object; M≥N≥3, and M and N are both integers;

[0032] Wherein, N checkerboard squares are respectively located at the vertices of the N corners of the N-gon surrounding the target object, and the remaining MN checkerboard squares are respectively located on the N sides of the N-gon surrounding the target object;

[0033] wherein the first image includes at least one checkerboard;

[0034] The second image includes at least one checkerboard.

[0035] In some embodiments, obtaining the actual three-dimensional coordinates corresponding to the reference point coordinates includes:

[0036] Establish a world coordinate system with the projection center of the target object on the ground as the origin of the world coordinate system;

[0037] Determining the actual three-dimensional coordinates of the corner points of the chessboard in the world coordinate system;

[0038] The determining of the external parameters of the camera to be calibrated based on the reference point coordinates, the actual three-dimensional coordinates, the internal parameters, and the distortion parameters includes:

[0039] Performing distortion correction on the reference point coordinates based on the internal parameters and the distortion parameters;

[0040] Based on the actual three-dimensional coordinates and the distortion-corrected coordinates of the reference point, the external parameters of the camera to be calibrated are obtained.

[0041] In some embodiments, adjusting the calibration object includes:

[0042] When calibrating a corresponding single camera to be calibrated using a single checkerboard, at least one of adjusting a size error of the checkerboard, adjusting the flatness of the ground, adjusting the flatness of the checkerboard cloth, and correcting an inclination of the checkerboard is adjusted;

[0043] and / or,

[0044] When calibrating a corresponding single camera to be calibrated using two checkerboards, at least one of adjusting the relative positions between the checkerboards, adjusting the spacing between the checkerboards, adjusting the flatness of the ground, and adjusting the flatness of the checkerboard cloth;

[0045] and / or,

[0046] When calibrating a corresponding single camera to be calibrated using at least three checkerboard squares, the site is changed to adjust the ground, and / or the layout of the checkerboard cloth is adjusted.

[0047] In some embodiments, the number of cameras to be calibrated is N;

[0048] When each of the cameras to be calibrated respectively acquires at least three consecutive checkerboard squares located on a single side of the N-gon, the method further includes:

[0049] Determining the back-projection error based on a single checkerboard grid, two checkerboard grids, and at least three checkerboard grids, respectively, and determining whether an adjustment condition is satisfied based on each back-projection error;

[0050] When any of the back-projection errors satisfies the adjustment condition, the calibration object is adjusted until the calibration ends when all the back-projection errors fail to satisfy the adjustment condition.

[0051] In some embodiments, determining that an adjustment condition is satisfied based on the back-projection error includes:

[0052] Determining whether the back-projection error is greater than or equal to a preset error threshold;

[0053] When the back-projection error is greater than or equal to the preset error threshold, it is determined that the back-projection error meets an adjustment condition.

[0054] In some embodiments, after determining whether the back-projection error is greater than or equal to a preset error threshold, the method further includes:

[0055] When the back-projection error is less than the preset error threshold, it is determined that the back-projection error does not meet the adjustment condition.

[0056] The present disclosure also provides a camera calibration device, which includes:

[0057] An image acquisition module, configured to acquire a first image captured by a camera to be calibrated; wherein the first image includes a reference calibration pattern;

[0058] A first determining module, configured to determine external parameters of the camera to be calibrated based on the first image;

[0059] a second determining module, configured to back-project the reference points in the reference calibration pattern based on the external parameters to determine a back-projection error;

[0060] a parameter adjustment module configured to adjust target parameters of the reference calibration pattern when an adjustment condition is determined to be satisfied based on the back projection error, and return to obtain a second image of the reference calibration pattern captured by the camera to be calibrated, including the reference calibration pattern after the target parameter adjustment, until the back projection error determined based on the second image no longer satisfies the adjustment condition, thereby completing the calibration.

[0061] In some embodiments, the camera to be calibrated has known intrinsic parameters and distortion parameters;

[0062] The first determining module is used to determine the external parameters of the camera to be calibrated based on the first image, and may specifically include:

[0063] determining coordinates of reference points in the reference calibration pattern based on the first image;

[0064] Obtaining the actual three-dimensional coordinates corresponding to the reference point coordinates;

[0065] Determining the extrinsic parameters of the camera to be calibrated based on the reference point coordinates, the actual three-dimensional coordinates, the intrinsic parameters, and the distortion parameters;

[0066] The second determining module is configured to perform back-projection on the reference points in the reference calibration pattern based on the external parameters to determine the back-projection error, which may specifically include:

[0067] Back-projecting to obtain calculated coordinates corresponding to the reference point coordinates based on the actual three-dimensional coordinates, the internal parameters, the distortion parameters, and the external parameters;

[0068] A back-projection error is determined based on the reference point coordinates and the corresponding calculated coordinates.

[0069] In some embodiments, the number of the reference point is at least one, the coordinates of the reference point are (x, y), and the calculated coordinates are (X, Y);

[0070] The second determining module is configured to determine the back-projection error based on the reference point coordinates and the corresponding calculated coordinates, and may specifically include:

[0071] The back-projection error is calculated using the following formula:

[0072]

[0073] Wherein, L represents the back-projection error, n represents the number of reference points, i represents the i-th reference point, n ≥ 1 and n is an integer, and i takes an integer value from 1 to n.

[0074] In some embodiments, the number of the cameras to be calibrated is at least one, and the number of the reference calibration patterns is at least one;

[0075] The second determining module is configured to determine a back projection error based on the reference point coordinates and the corresponding calculated coordinates, and may specifically include:

[0076] When the number of the reference calibration patterns captured by each camera to be calibrated is j, determining a back-projection error based on the coordinates of the reference points in the m reference calibration patterns and the corresponding calculated coordinates;

[0077] Wherein, j≥1 and j is an integer, and m takes integer values from 1 to j in sequence.

[0078] In some embodiments, the reference calibration pattern comprises a checkerboard, and the reference point in the reference calibration pattern comprises at least one corner point of the checkerboard;

[0079] The reference calibration pattern is set on a calibration object; correspondingly, the calibration object includes a checkerboard cloth with checkerboard patterns or a ground with checkerboard patterns;

[0080] The parameter adjustment module is used to adjust the target parameters of the reference calibration pattern, which may specifically include:

[0081] The calibration object is adjusted.

[0082] In some embodiments, the calibration object includes M checkerboards surrounding the target object; M≥N≥3, and M and N are both integers;

[0083] Wherein, N checkerboard squares are respectively located at the vertices of the N corners of the N-gon surrounding the target object, and the remaining MN checkerboard squares are respectively located on the N sides of the N-gon surrounding the target object;

[0084] wherein the first image includes at least one checkerboard;

[0085] The second image includes at least one checkerboard.

[0086] In some embodiments, the first determining module is configured to obtain the actual three-dimensional coordinates corresponding to the reference point coordinates, which may specifically include:

[0087] Establish a world coordinate system with the projection center of the target object on the ground as the origin of the world coordinate system;

[0088] Determining the actual three-dimensional coordinates of the corner points of the chessboard in the world coordinate system;

[0089] The first determination module is configured to determine the external parameters of the camera to be calibrated based on the reference point coordinates, the actual three-dimensional coordinates, the internal parameters, and the distortion parameters, and may specifically include:

[0090] Performing distortion correction on the reference point coordinates based on the internal parameters and the distortion parameters;

[0091] Based on the actual three-dimensional coordinates and the distortion-corrected coordinates of the reference point, the external parameters of the camera to be calibrated are obtained.

[0092] In some embodiments, the parameter adjustment module is used to adjust the calibration object, which may specifically include:

[0093] When calibrating a corresponding single camera to be calibrated using a single checkerboard, at least one of adjusting a size error of the checkerboard, adjusting the flatness of the ground, adjusting the flatness of the checkerboard cloth, and correcting an inclination of the checkerboard is adjusted;

[0094] and / or,

[0095] When calibrating a corresponding single camera to be calibrated using two checkerboards, at least one of adjusting the relative positions between the checkerboards, adjusting the spacing between the checkerboards, adjusting the flatness of the ground, and adjusting the flatness of the checkerboard cloth;

[0096] and / or,

[0097] When calibrating a corresponding single camera to be calibrated using at least three checkerboard squares, the site is changed to adjust the ground, and / or the layout of the checkerboard cloth is adjusted.

[0098] In some embodiments, the number of cameras to be calibrated is N;

[0099] When each of the cameras to be calibrated respectively acquires at least three consecutive checkerboard squares located on a single side of the N-gon, the apparatus further includes:

[0100] an auxiliary judgment module, configured to determine the back-projection error based on a single checkerboard grid, two checkerboard grids, and at least three checkerboard grids, respectively, and to judge whether an adjustment condition is satisfied based on each back-projection error;

[0101] The auxiliary adjustment module is used to adjust the calibration object when any of the back-projection errors meets the adjustment condition, and the calibration is completed when all the back-projection errors do not meet the adjustment condition.

[0102] In some embodiments, the parameter adjustment module is used to determine whether an adjustment condition is satisfied based on the back projection error, which may specifically include:

[0103] Determining whether the back-projection error is greater than or equal to a preset error threshold;

[0104] When the back-projection error is greater than or equal to the preset error threshold, it is determined that the back-projection error meets an adjustment condition.

[0105] In some embodiments, the parameter adjustment module is further configured to:

[0106] When the back-projection error is less than the preset error threshold, it is determined that the back-projection error does not meet the adjustment condition.

[0107] The present disclosure also provides an image stitching method, which includes:

[0108] Acquiring images to be stitched using at least two cameras;

[0109] Calibrate at least two cameras using any of the above camera calibration methods;

[0110] Based on the camera parameters determined by calibration, the images to be stitched are stitched.

[0111] The present disclosure further provides an image stitching device, comprising:

[0112] A module for acquiring images to be stitched, configured to acquire images to be stitched using at least two cameras;

[0113] An image calibration module, configured to calibrate at least two cameras using any of the above-mentioned camera calibration methods;

[0114] The image stitching module is used to stitch the images to be stitched based on the camera parameters determined by calibration.

[0115] The present disclosure further provides a surround-view camera, which calibrates at least two cameras using any of the above-mentioned camera calibration methods; or

[0116] Use any of the above image stitching methods to stitch the surround view images.

[0117] The present disclosure also provides a vehicle, which includes any one of the above-mentioned surround-view cameras.

[0118] The technical solution provided by the embodiments of the present disclosure has the following advantages over the prior art:

[0119] The camera calibration method provided by the embodiments of the present disclosure includes: obtaining a first image captured by a camera to be calibrated; wherein the first image includes a reference calibration pattern; determining external parameters of the camera to be calibrated based on the first image; back-projecting reference points in the reference calibration pattern based on the external parameters to determine a back-projection error; when it is determined based on the back-projection error that an adjustment condition is satisfied, adjusting target parameters of the reference calibration pattern, and returning to obtain a second image captured by the camera to be calibrated and including the reference calibration pattern after the target parameters are adjusted, until the back-projection error determined based on the second image does not satisfy the adjustment condition, at which point the calibration is completed. Therefore, in the camera calibration method, when the back projection error meets the adjustment conditions, the calibration accuracy of the corresponding camera's external parameters is low; in response to this, by adjusting the target parameters of the reference calibration pattern and looping the calibration process until the back projection error does not meet the adjustment conditions, the calibration accuracy of the corresponding camera's external parameters is high, thereby improving the problem of large back projection error caused by objective factors of the reference calibration pattern itself (the reference calibration pattern can be set on the calibration object, for example), thereby improving the problem of low calibration accuracy of the camera's external parameters caused by objective factors of the calibration object itself; thereby improving the calibration accuracy of the camera's external parameters, which is beneficial to reducing the stitching error of image stitching, and further beneficial to improving the image stitching effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0120] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0121] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0122] Figure 1 A schematic diagram of a camera calibration method according to an embodiment of the present disclosure;

[0123] Figure 2 A schematic structural diagram of a calibration object provided in an embodiment of the present disclosure;

[0124] Figure 3 A schematic structural diagram of a camera calibration device provided in an embodiment of the present disclosure;

[0125] Figure 4 A flowchart of an image stitching method provided by an embodiment of the present disclosure;

[0126] Figure 5A schematic structural diagram of an image stitching device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0127] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0128] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0129] In the embodiments of the present disclosure, key terms are first explained.

[0130] The intrinsic parameters of a camera, also known as internal parameters or internal parameters, are parameters related to the camera's own characteristics, such as the camera's focal length, pixel size, and other parameters; they are generally calculated and represented in the form of a 3*3 intrinsic parameter matrix.

[0131] The camera's extrinsic parameters, also known as external parameters or external parameters, are parameters in the world coordinate system, such as the camera's position and rotation direction. They are generally calculated and represented in the form of a 4*4 extrinsic parameter matrix.

[0132] The camera calibration method and image stitching method provided in the embodiments of the present disclosure can be applied to a vehicle-mounted panoramic view system, for example, in assisted driving; they can also be applied to other panoramic view scenarios, which are not limited here.

[0133] Taking the on-board panoramic view system as an example, in the camera calibration process of the on-board panoramic view system, when the checkerboard is used as a reference calibration pattern, the checkerboard can be drawn directly on the ground, or the checkerboard can be printed on a checkerboard cloth of relatively good quality (i.e., scene calibration cloth), and then the checkerboard cloth is spread on the ground as needed, and the camera is used to obtain an image including the checkerboard pattern, and the camera's external parameters are calibrated based on the obtained image, thereby achieving the registration between the video images obtained by 4-8 cameras installed around the vehicle body, and achieving the purpose of panoramic view stitching. In this process, in engineering, due to the influence of objective factors of the calibration object itself, such as the process of drawing or printing the checkerboard, the size error between each checkerboard, the wrinkles of the checkerboard cloth, the flatness of the ground, etc., it may be impossible to calibrate the appropriate external parameters, that is, the calibration accuracy of the external parameters is poor, which in turn leads to large image stitching errors and poor stitching effects.

[0134] To this end, the embodiments of the present disclosure provide a camera calibration method, which can reduce the influence of the objective factors of the calibration object itself on the calibration accuracy of the camera extrinsic parameters by adjusting and improving at least one of the above-mentioned objective factors, thereby improving the problem of poor calibration accuracy of the camera extrinsic parameters, that is, it can improve the calibration accuracy of the camera extrinsic parameters, and further reduce the image stitching error and improve the image stitching effect.

[0135] The following combination Figure 1-Figure 5 , the camera calibration method, image stitching method, device, surround view camera and vehicle provided by the embodiments of the present disclosure are exemplarily described.

[0136] S110: Obtain a first image captured by the camera to be calibrated.

[0137] The first image includes a reference calibration pattern.

[0138] The reference calibration pattern is a pattern with reference points used to calibrate camera extrinsic and intrinsic parameters. The reference calibration pattern can be captured by the camera to be calibrated. For example, the reference calibration pattern may include a checkerboard or other patterns known to those skilled in the art, which are not limited here.

[0139] Exemplarily, before this step, a calibration object having a reference calibration pattern may be provided, and a first image including the reference calibration pattern may be captured using a camera; correspondingly, the camera calibration device may obtain the first image and perform subsequent S120-S140 based on the first image to achieve calibration of the camera extrinsic parameters.

[0140] Among them, the reference calibration pattern can be set on the calibration object, and the calibration object with the reference calibration pattern is a movable or immovable object used to calibrate the camera, for example, it can be one, two or more scene calibration cloths printed with a checkerboard pattern, or it can be a ground with a checkerboard pattern, and it can be other calibration objects known to those skilled in the art, which are not limited here.

[0141] For example, Figure 2 FIG. 1 shows a schematic diagram of the structure of a calibration object provided by an embodiment of the present disclosure. Figure 2 As shown, the reference calibration pattern in the calibration object is a checkerboard pattern, represented by first checkerboard 201, second checkerboard 202, third checkerboard 203, fourth checkerboard 204, fifth checkerboard 205, sixth checkerboard 206, seventh checkerboard 207, and eighth checkerboard 208. In an actual structure, the calibration object can be a ground with a checkerboard pattern, or a checkerboard cloth printed with a checkerboard pattern and spread on the ground, which is not limited here.

[0142] In the disclosed embodiment, a checkerboard pattern may be drawn on the ground, or a checkerboard cloth printed with checkerboard patterns may be spread on the ground, or other methods known to those skilled in the art may be used to provide a reference calibration pattern, or other calibration objects having other reference calibration patterns may be provided, which are not limited here.

[0143] S120: Determine external parameters of the camera to be calibrated based on the first image.

[0144] Among them, the first image is a two-dimensional image in the image plane coordinate system. By performing coordinate conversion between it and the corresponding points in the camera coordinate system and the world coordinate system, the external parameters of the camera to be calibrated can be determined.

[0145] In some embodiments, the camera to be calibrated has known intrinsic parameters and distortion parameters.

[0146] The intrinsic parameters and distortion parameters of the camera to be calibrated can be pre-calibrated and used as known parameters in the camera extrinsic parameter calibration process provided by the embodiments of the present disclosure. For example, before executing the camera calibration method provided by the embodiments of the present disclosure, the camera's intrinsic parameters and distortion parameters can be calibrated first, and the calibrated intrinsic parameters and distortion parameters can be used in the method.

[0147] Based on this, the step of "determining the external parameters of the camera to be calibrated based on the first image" may specifically include:

[0148] determining coordinates of reference points in a reference calibration pattern based on the first image;

[0149] Get the actual three-dimensional coordinates corresponding to the reference point coordinates;

[0150] The extrinsic parameters of the camera to be calibrated are determined based on the reference point coordinates, actual 3D coordinates, intrinsic parameters, and distortion parameters.

[0151] The first image includes a reference calibration pattern, the reference calibration pattern having reference points, and the reference points correspond to reference point coordinates in the image coordinate system and actual three-dimensional coordinates in the world coordinate system. For example, if the reference calibration pattern is a checkerboard, the reference points may be corner points of the checkerboard.

[0152] In the embodiment of the present disclosure, the external parameters of the camera to be calibrated can be solved based on the reference point coordinates and the actual three-dimensional coordinates using any method known to those skilled in the art.

[0153] It is understandable that if the internal parameters of the camera to be calibrated are unknown, before this step, any method known to those skilled in the art can be used to solve the internal parameters of the camera to be calibrated; and when the camera to be calibrated is a fisheye camera or other camera with distortion, before this step, any method known to those skilled in the art can be used to solve the distortion parameters of the camera to be calibrated.

[0154] S130 : Back-project the reference points in the reference calibration pattern based on the external parameters to determine a back-projection error.

[0155] The extrinsic parameters can be understood as the parameters obtained based on the forward solution process. In this step, the extrinsic parameters are taken as known, and the actual 3D coordinates corresponding to the reference points in the reference calibration pattern are converted to 2D coordinates. The error between the calculated coordinates obtained by the conversion and the coordinates of the reference points is determined, which is the back-projection error.

[0156] In some embodiments, this step may specifically include:

[0157] Based on the actual three-dimensional coordinates, internal parameters, distortion parameters and external parameters, back-projection is performed to obtain calculated coordinates corresponding to the coordinates of the reference point;

[0158] Based on the reference point coordinates and the corresponding calculated coordinates, the back-projection error is determined.

[0159] Among them, back projection refers to the process of converting three-dimensional actual coordinates into two-dimensional image coordinates, that is, calculated coordinates corresponding to the reference point coordinates, through external parameter and internal parameter transformation.

[0160] In this way, the actual 3D coordinates can be mapped to the camera coordinate system using the extrinsic matrix to obtain the corresponding coordinates in the camera coordinate system. Combined with the intrinsic matrix, the corresponding coordinates in the camera coordinate system are mapped to the image coordinate system to obtain the calculated coordinates corresponding to the reference point coordinates. Alternatively, the actual 3D coordinates can be directly mapped to the image coordinate system using the extrinsic and intrinsic matrices to obtain the calculated coordinates corresponding to the reference point coordinates, which is not limited here.

[0161] Based on this, the reference point coordinates are the coordinates of the reference point in the image coordinate system determined based on the first image, and the calculated coordinates are the calculated coordinates corresponding to the reference point coordinates obtained by back-projection. Due to the influence of objective factors such as the reference calibration pattern and the calibration object itself, there may be differences between the actual 3D coordinates and the theoretical 3D coordinates of the corresponding reference point, which in turn may result in low calibration accuracy of the camera's extrinsic parameters. The reference point coordinates and the corresponding calculated coordinates may not be completely consistent. The greater the difference between the two, the lower the calibration accuracy of the camera's extrinsic parameters. Therefore, by calculating the difference between the two and expressing it as a back-projection error, it is easier to subsequently determine whether the calibration accuracy of the camera's extrinsic parameters meets the standards.

[0162] S140. When it is determined based on the back projection error that the adjustment condition is met, the target parameters of the reference calibration pattern are adjusted, and the second image of the reference calibration pattern after the target parameter adjustment, which is captured by the camera to be calibrated, is obtained. The calibration is completed until the back projection error determined based on the second image does not meet the adjustment condition.

[0163] The adjustment condition corresponds to the condition that the extrinsic calibration accuracy of the camera to be calibrated does not meet the standard. Therefore, when the back-projection error meets the adjustment condition, it indicates that the extrinsic calibration accuracy of the camera to be calibrated does not meet the standard. In this case, it is necessary to adjust the target parameters of the reference calibration pattern. Based on the adjusted reference calibration pattern, the above S110 is executed again, and the process of S110-S140 is repeated until the back-projection error no longer meets the adjustment condition, that is, the extrinsic calibration accuracy of the camera to be calibrated meets the standard, and the calibration is completed.

[0164] It can be understood that both the first image and the second image correspond to images with a reference calibration pattern captured by the camera, the only difference being that the first image corresponds to the image before the target parameters of the reference calibration pattern are adjusted, and the second image corresponds to the image after the target parameters of the reference calibration pattern are adjusted.

[0165] Exemplarily, the target parameter of the reference calibration pattern may be the position of a reference point in the reference calibration pattern or other parameters that affect the extrinsic parameter calibration, which is not limited here.

[0166] For example, taking the target parameter of the reference calibration pattern as the position of the reference point therein as an example, in order to improve the accuracy of the external parameter calibration, based on the distribution regularity of the positions of the reference points of the reference calibration pattern, the positions of one or several reference points that do not meet the distribution regularity can be manually adjusted or automatically adjusted using a computer program, so as to use the adjusted reference calibration pattern to perform external parameter calibration and improve the accuracy of the external parameter calibration.

[0167] Alternatively, a specific implementation method for adjusting the target parameters of the reference calibration pattern may include: a camera extrinsic calibration device generates an adjustment instruction and transmits the adjustment instruction to a motion mechanism, such as a robotic arm; the motion mechanism may adjust the calibration object in response to the adjustment instruction, thereby achieving adjustment of the target parameters of the reference calibration pattern.

[0168] In other embodiments, the target parameters of the reference calibration pattern may be adjusted in other ways known to those skilled in the art, which are not limited here.

[0169] In the camera calibration method provided by the embodiments of the present disclosure, a first image including a reference calibration pattern is acquired by the camera to be calibrated; external parameters of the camera to be calibrated are determined based on the first image; reference points in the reference calibration pattern are back-projected based on the external parameters to determine a back-projection error; when an adjustment condition is determined to be satisfied based on the back-projection error, target parameters of the reference calibration pattern are adjusted, and a second image including the reference calibration pattern after the target parameters are adjusted is acquired by the camera to be calibrated, and the calibration is completed until the back-projection error determined based on the second image does not satisfy the adjustment condition. Among them, when the back projection error meets the adjustment conditions, the external parameter calibration accuracy of the corresponding camera is low; in response to this, by adjusting the target parameters of the reference calibration pattern and looping the calibration process until the back projection error does not meet the adjustment conditions, the external parameter calibration accuracy of the corresponding camera is high, thereby improving the problem of large back projection error caused by objective factors of the reference calibration pattern itself (the reference calibration pattern can be set on the calibration object, for example), thereby improving the problem of low calibration accuracy of the camera's external parameters caused by objective factors of the calibration object itself; thereby improving the calibration accuracy of the camera's external parameters, which is beneficial to reducing the stitching error of image stitching, and further beneficial to improving the image stitching effect.

[0170] In addition, the calibration object adjusted in this camera calibration method can also be applied to the calibration process of other cameras to be calibrated, so as to reduce the influence of the objective factors of the calibration object itself on the calibration accuracy, thereby achieving better calibration results, saving calibration time and improving calibration efficiency.

[0171] In some embodiments, the number of reference points is at least one, the reference point coordinates are (x, y), and the calculated coordinates are (X, Y);

[0172] Determining the back-projection error based on the reference point coordinates and the corresponding calculated coordinates includes:

[0173] The back-projection error is calculated using the following formula:

[0174]

[0175] Wherein, L represents the back-projection error, n represents the number of reference points, i represents the i-th reference point, n ≥ 1 and n is an integer, and i takes an integer value from 1 to n.

[0176] The distance between each reference point coordinate and its corresponding calculated coordinate can be calculated, and the average distance can be calculated to obtain the back-projection error. Thus, the back-projection error can be calculated based on all reference point coordinates and their corresponding calculated coordinates, which can more accurately characterize the difference between the reference point coordinates and the calculated coordinates, and thus more accurately characterize the calibration accuracy of the camera's extrinsic parameters, making it easier to accurately determine whether the calibration accuracy of the camera's extrinsic parameters meets the standards, thereby helping to improve the calibration accuracy of the camera's extrinsic parameters.

[0177] Exemplarily, the value of n is 20, and i is any integer value from 1 to 20. In other embodiments, the value of n may be any other positive integer value, which may be set based on camera calibration requirements and is not limited here.

[0178] In some embodiments, the number of cameras to be calibrated is at least one, and the number of reference calibration patterns is at least one.

[0179] In combination with the above, when the number of reference calibration patterns collected by each camera to be calibrated is j, determining the back-projection error based on the reference point coordinates and the corresponding calculated coordinates may specifically include:

[0180] determining a back-projection error based on coordinates of reference points in the m reference calibration patterns and corresponding calculated coordinates;

[0181] Wherein, j≥1 and j is an integer, and m takes integer values from 1 to j in sequence.

[0182] In the embodiment of the present disclosure, when the number of reference calibration patterns is at least one, the more reference calibration patterns used in determining the back-projection error, the more accurately the difference between the reference point coordinates and the calculated coordinates can be represented, that is, the more accurately the calibration accuracy of the camera's external parameters can be represented, which is conducive to accurately judging whether its calibration accuracy meets the standards and is conducive to improving the calibration accuracy of the camera's external parameters.

[0183] In some embodiments, when the number of reference calibration patterns is at least one, the aforementioned S102-S108 can be performed sequentially using 1, 2, ..., j reference calibration patterns, so as to adjust the objective factors of the calibration object in a targeted manner and gradually improve the calibration accuracy of the camera's external parameters in steps. An example explanation will be given below.

[0184] In some embodiments, continue to refer to Figure 2The reference calibration pattern includes a checkerboard, and the reference point in the reference calibration pattern includes at least one corner point of the checkerboard; the calibration object with the reference calibration pattern includes a checkerboard cloth with checkerboard or a ground with checkerboard drawn on it.

[0185] In some embodiments, the reference calibration pattern includes a checkerboard, and the reference points in the reference calibration pattern include at least one corner point of the checkerboard; the reference calibration pattern is set on the calibration object; correspondingly, the calibration object includes a checkerboard cloth with checkerboards or a ground with checkerboards drawn on it.

[0186] Therefore, the above-mentioned “providing a calibration object having a reference calibration pattern” may specifically include:

[0187] Arrange a checkerboard cloth with checkerboard patterns around the marked position of the target object; or

[0188] Draw a checkerboard on the ground around the target's calibrated position;

[0189] The target object carries the camera to be calibrated.

[0190] For example, the target object is located at Figure 2 200 is shown as the position of the central rectangular area; taking the on-board panoramic view system as an example, the target object calibration position 200 can be the position where the vehicle is parked, that is, the car occupancy position. A checkerboard is set around the target object calibration position 200 represented by the central rectangular area. For example, the checkerboard can be drawn on the ground around the target object calibration position 200 by spraying, tracing, etc.; or when the checkerboard is printed on a checkerboard cloth, the checkerboard cloth can be arranged according to the Figure 2 The positions of the checkerboard are shown to be arranged around the circumference of the target's marked position 200 . Figure 2 The 8 checkerboard squares shown can be located on the same checkerboard cloth or on multiple different checkerboard cloths. The number of checkerboard squares on each checkerboard cloth can be 1, 2 or more, and can be the same or different, which is not limited here.

[0191] Based on this, in some embodiments, “adjusting target parameters of the reference calibration pattern” may specifically include:

[0192] Adjust the calibration object.

[0193] That is, by adjusting the calibration object, the target parameters of the reference calibration pattern are adjusted.

[0194] In some embodiments, the calibration object includes M checkerboards surrounding the target object; M≥N≥3, and M and N are both integers; wherein the N checkerboards are respectively located at the vertex positions of the N corners of the N-gon surrounding the target object, and the remaining MN checkerboards are respectively located on the N sides of the N-gon surrounding the target object; wherein the first image includes at least one checkerboard; and the second image includes at least one checkerboard.

[0195] In this way, the calibration object and the reference calibration pattern can be arranged around the target object, and the images captured by the camera to be calibrated, including the first image and the second image, both include at least one checkerboard.

[0196] For example, continue to refer to Figure 2 , the calibration object includes 8 checkerboard grids surrounding the target object; among them, 4 checkerboard grids (indicated by the first checkerboard grid 201, the third checkerboard grid 203, the fifth checkerboard grid 205 and the seventh checkerboard grid 207) are respectively located at the vertices of the four corners of the quadrilateral surrounding the target object, and the remaining 4 checkerboard grids (indicated by the second checkerboard grid 202, the fourth checkerboard grid 204, the sixth checkerboard grid 206 and the eighth checkerboard grid 208) are respectively located on the four sides of the quadrilateral surrounding the target object.

[0197] For example, the checkerboard grids on the long sides of the quadrilateral, i.e., the fourth checkerboard grid 204 and the eighth checkerboard grid 208, are larger checkerboard grids, for example, the length of a single side of the square grids may be 50 cm; the remaining checkerboard grids are smaller checkerboard grids, for example, the length of a single side of the square grids may be 25 cm.

[0198] Exemplarily, the intervals between adjacent checkerboard squares may be set as follows: the first interval D11 and the second interval D12 are the same, both 140 cm; the third interval D21 and the fourth interval D22 are the same, both 90 cm; the fifth interval D31 and the sixth interval D32 are the same, both 90 cm; the seventh interval D41 and the eighth interval D42 are the same, both 80 cm.

[0199] For example, in combination Figure 2 , the image captured by the camera to be calibrated may be an image including three checkerboard squares located on the same side.

[0200] In other embodiments, the number and arrangement of checkerboard grids, the number, size and arrangement of square grids in each checkerboard grid, the relative position relationship and spacing between adjacent checkerboard grids can all be set according to the camera calibration requirements. In effect, the checkerboard grids can be set around the target calibration position 200. Checkerboard grids of different sizes in the left, right, front and back directions set around the target calibration position 200 can be used, ensuring that each camera to be calibrated can capture at least one checkerboard grid, for example, 3 checkerboard grids, and can identify the corner points of the checkerboard grids. It can be adjusted according to the size of the vehicle body and the posture and angle of the camera installed on the vehicle body to meet the above-mentioned effect requirements. It will not be repeated here and is not limited.

[0201] In some embodiments, the above “obtaining actual three-dimensional coordinates corresponding to the reference point coordinates” may specifically include:

[0202] Establish a world coordinate system with the projection center of the target object on the ground as the origin of the world coordinate system;

[0203] Determine the actual 3D coordinates of the corner points of the chessboard in the world coordinate system.

[0204] For example, Figure 2 Taking the orientation shown as an example and extending upward in a direction perpendicular to the plane, the center of the vehicle's projection on the ground can be used as the origin of the world coordinate system, that is, the center of the rectangular area of the target object calibration position 200 can be used as the origin of the world coordinate system, the right is the X-axis direction, the forward is the Y-axis direction, and the upward is the Z-axis direction. A world coordinate system is established, and the three-dimensional coordinates of each corner point of the calibration object (i.e., the chessboard) in the world coordinate system, i.e., the actual three-dimensional coordinates, are obtained.

[0205] In some embodiments, the above “determining the extrinsic parameters of the camera to be calibrated based on the reference point coordinates, the actual three-dimensional coordinates, the intrinsic parameters, and the distortion parameters” may specifically include:

[0206] Based on the internal parameters and distortion parameters, the reference point coordinates are corrected for distortion;

[0207] Based on the actual 3D coordinates and the coordinates of the reference point after distortion correction, the extrinsic parameters of the camera to be calibrated are obtained.

[0208] For example, taking a vehicle as an example, the vehicle body is arranged on the flat ground around the vehicle body. Figure 2The checkerboard pattern shown above forms eight checkerboard squares with a defined positional relationship, which form the calibration object. A world coordinate system is established with the center of the vehicle's projection on the ground as the origin, and the actual 3D coordinates of each corner point of the calibration object in the world coordinate system are obtained. Each camera to be calibrated acquires an image with the checkerboard squares and automatically extracts the corner points of the three checkerboard squares (including the left, center, and right checkerboard squares in each image). The extrinsic parameters of each camera are calculated based on the coordinates (x, y) of the corner points, or reference points (which are distortion-corrected coordinates calculated from the intrinsic distortion parameters and the original corner point coordinates), and the actual 3D coordinates of each corner point of the calibration object in the world coordinate system.

[0209] In the disclosed embodiment, when the camera to be calibrated has distortion, the external parameters of each camera to be calibrated can be solved based on the coordinates of the reference point after distortion correction and the actual three-dimensional coordinates of the corresponding reference point in the world coordinate system. The solution accuracy of the external parameters is high, which is conducive to improving the calibration accuracy of the external parameters.

[0210] In some embodiments, the “adjustment calibration object” mentioned above may specifically include:

[0211] When calibrating a corresponding single camera to be calibrated using a single checkerboard, at least one of adjusting a size error of the checkerboard, adjusting the flatness of the ground, adjusting the flatness of the checkerboard cloth, and correcting an inclination of the checkerboard is adjusted;

[0212] and / or,

[0213] When calibrating a corresponding single camera to be calibrated using two checkerboards, at least one of adjusting the relative positions between the checkerboards, adjusting the spacing between the checkerboards, adjusting the flatness of the ground, and adjusting the flatness of the checkerboard cloth is adjusted;

[0214] and / or,

[0215] When calibrating a corresponding single camera to be calibrated using at least three checkerboard squares, the site is changed to adjust the ground surface, and / or the chessboard cloth is adjusted.

[0216] When calibrating a single camera to be calibrated using a single checkerboard, “adjusting the calibration object” may specifically include:

[0217] At least one of adjusting the size error of the checkerboard, adjusting the flatness of the ground, adjusting the flatness of the checkerboard cloth, and correcting the inclination of the checkerboard is provided.

[0218] In the disclosed embodiment, the external parameters of a single camera can be calculated based on the corner points of a single chessboard, and its back-projection error can be calculated. When the back-projection error is large, that is, the back-projection error is equal to or greater than a preset error threshold, the objective factors of the calibration object itself, such as the size error of the square grid of the chessboard, whether the ground is flat, whether the chessboard cloth is wrinkled, and whether the grid is skewed, can be checked. At least one of the above factors can be changed. For example, if the size error of the chessboard is large, the chessboard cloth can be replaced or the chessboard can be redrawn on the ground; if the ground flatness is poor, the venue can be changed or the ground can be leveled to improve the ground flatness; if the flatness of the chessboard cloth is poor, for example, if the chessboard cloth is heavily wrinkled, the chessboard cloth can be re-laid or replaced with a chessboard cloth with higher precision; if the grid of the chessboard is tilted, that is, if the tilt of the chessboard is large, the chessboard cloth can be replaced or the chessboard can be redrawn.

[0219] Thereafter, the extrinsic parameters are recalculated based on the corner points of the single chessboard until the obtained back-projection error is less than a preset error threshold.

[0220] Optionally, each checkerboard block is adjusted in the above manner so that the camera extrinsic parameters obtained based on the calibration of each single checkerboard block and the back projection errors calculated using the camera extrinsic parameters are all smaller than a preset error threshold.

[0221] In this way, the influence of the objective factors of the calibration object itself on the camera calibration can be reduced, and the calibration accuracy of the camera extrinsic parameters can be improved.

[0222] When calibrating a single camera to be calibrated using two checkerboards, “adjusting the calibration object” may specifically include:

[0223] At least one of adjusting the relative positions of the checkerboard squares, adjusting the spacing between the checkerboard squares, adjusting the flatness of the ground, and adjusting the flatness of the checkerboard cloth.

[0224] In the disclosed embodiment, calibration can be performed directly, or after calibration is performed using the corner points of a single checkerboard, the corner points of two adjacent checkerboards can be selected. The camera's extrinsic parameters and back-projection error can be calculated based on the selected corner points. When the back-projection error is too large, that is, the back-projection error is greater than or equal to a preset error threshold, objective factors of the calibration object itself, such as whether the spacing between the two checkerboards is accurate, whether the ground is flat, whether the checkerboard cloth is wrinkled, and whether there is any deviation in the relative position, can be checked. At least one of the above factors can be modified. For example, if the relative position and spacing between the two checkerboards deviate from the preset values, they can be adjusted to the preset values; if the ground is not flat, the site can be changed or the ground can be leveled to improve the flatness; if the checkerboard cloth is not flat, such as if it is heavily wrinkled, the checkerboard cloth can be re-laid or replaced with a higher-precision checkerboard cloth. Subsequently, the extrinsic parameters can be recalculated based on the corner points of the two adjacent checkerboards until the back-projection error is less than the preset error threshold.

[0225] When calibrating a single camera to be calibrated using three checkerboards, “adjusting the calibration object” may specifically include:

[0226] Change the playing surface and / or adjust the chessboard layout.

[0227] In the embodiment of the present disclosure, it is possible to directly, or after calibration using the corner points of two adjacent chessboard squares, use the corner points of three chessboard squares to calculate external parameters, use the calculated external parameters to back-project the actual three-dimensional coordinates, and calculate the back-projection error. When the back-projection error is too large, that is, the back-projection error is greater than or equal to a preset error threshold, the venue can be changed, for example, to a flatter venue, and / or the laid chessboard cloth can be adjusted, for example, by changing the chessboard cloth, laying the chessboard cloth more flat, or adjusting at least one of the orientation and spacing between the chessboard cloths, until the corresponding back-projection error is less than the preset error threshold when calculating the external parameters using the corner points of three chessboard squares.

[0228] It is understandable that a chessboard cloth may be printed with one, two or more chessboard squares, and the relative positions of adjacent chessboard squares on different chessboard cloths must be adjusted by adjusting the relative positions of the chessboard cloths.

[0229] In some embodiments, the number of cameras to be calibrated is N; optionally, N=4; when each camera to be calibrated acquires at least three consecutive checkerboard squares located on a single side of an N-sided polygon, the method may further include:

[0230] Determining back-projection errors based on a single checkerboard grid, two checkerboard grids, and at least three checkerboard grids, respectively, and determining whether an adjustment condition is satisfied based on each back-projection error;

[0231] When any back-projection error satisfies the adjustment condition, the calibration object is adjusted until all back-projection errors fail to satisfy the adjustment condition, and the calibration is completed.

[0232] In the disclosed embodiment, the extrinsic parameters of each individual camera can be calculated based on the corner points of a single checkerboard grid, and the back-projection error can be calculated. When the back-projection error is equal to or greater than a preset error threshold, objective factors of the calibration object, such as the size error of the square grid, the flatness of the ground, the wrinkles of the checkerboard, and the skewness of the grid, can be checked. After changing at least one of these factors, the extrinsic parameters are recalculated based on the corner points of the single checkerboard grid until the back-projection error is less than the preset error threshold.

[0233] Next, the camera's extrinsic parameters are calculated based on the corner points of two adjacent checkerboard squares, along with the back-projection error. If the back-projection error is greater than or equal to a preset error threshold, the camera can be inspected for objective factors such as the accuracy of the spacing between the two checkerboard squares, the flatness of the ground, the wrinkles on the checkerboard cloth, and any deviations in relative position. After modifying at least one of these factors, the extrinsic parameters are recalculated based on the corner points of the two adjacent checkerboard squares until the back-projection error is less than the preset error threshold.

[0234] The three checkerboard corner points are then used to calculate extrinsic parameters. The calculated extrinsic parameters are then used to back-project the actual 3D coordinates, and the back-projection error is calculated. If the back-projection error is greater than or equal to a preset error threshold, the playing area can be changed and / or the chessboard cloth can be adjusted until the back-projection error is less than the preset error threshold when the extrinsic parameters are calculated using the three checkerboard corner points.

[0235] Therefore, four cameras are calibrated, and the objective factors of other calibration objects such as the calibration cloth and the site are adjusted through quantitative back-projection errors to improve the camera calibration accuracy, which is beneficial to improving the image stitching effect.

[0236] For example, in the process of extrinsic calibration of cameras in a vehicle-mounted panoramic surround view system, if one camera is installed on each of the front, rear, left, and right sides of the vehicle body, that is, a total of four cameras, the camera calibration method provided by the embodiment of the present disclosure can be used to calibrate the cameras, which can quantitatively judge and improve the calibration accuracy, thereby improving the surround view stitching error.

[0237] In other embodiments, the number of cameras is not limited, and the number of checkerboard squares that each camera can capture is also not limited. Using only one checkerboard square to reduce back-projection error can also achieve the effect of improving calibration accuracy. Optionally, using two checkerboard squares to reduce back-projection error can further improve calibration accuracy. Optionally, using three checkerboard squares to reduce back-projection error can further improve calibration accuracy. To achieve higher calibration accuracy, more checkerboard squares can be used, which is not limited here.

[0238] In some embodiments, determining that an adjustment condition is satisfied based on the back-projection error includes:

[0239] Determining whether the back-projection error is greater than or equal to a preset error threshold;

[0240] When the back-projection error is greater than or equal to the preset error threshold, it is determined that the back-projection error meets the adjustment condition.

[0241] The preset error threshold serves as a reference value for measuring whether the calibration accuracy of the camera's extrinsic parameters meets the standards. Based on the above, the smaller the backprojection error, the higher the calibration accuracy of the camera's extrinsic parameters. Therefore, if the backprojection error is less than the preset error threshold, the calibration accuracy of the camera's extrinsic parameters meets the standards. Otherwise, if the backprojection error is greater than or equal to the preset error threshold, the calibration accuracy of the camera's extrinsic parameters does not meet the standards. Based on this, the subsequent steps are executed.

[0242] For example, the preset error threshold may be any value set based on the calibration accuracy of the camera's external parameters, and may be set based on camera calibration requirements, which is not limited herein.

[0243] Among them, combined with the above, when the back projection error is greater than or equal to the preset error threshold, it indicates that the difference between the reference point coordinates and the corresponding calculated coordinates is large, that is, the calibration accuracy of the camera's external parameters is low and cannot meet the camera's calibration requirements. At this time, if the adjustment conditions are met, the calibration object needs to be adjusted to reduce the impact of the objective factors of the calibration object and the reference calibration pattern set on it on the camera calibration.

[0244] In some embodiments, after determining whether the back-projection error is greater than or equal to a preset error threshold, the method further includes:

[0245] When the back-projection error is less than the preset error threshold, it is determined that the back-projection error does not meet the adjustment condition.

[0246] Among them, combined with the above, when the back projection error is less than the preset error threshold, it indicates that the difference between the reference point coordinates and the corresponding calculated coordinates is small, that is, the calibration accuracy of the camera's external parameters is high, which can meet the camera's calibration requirements and no further adjustment is required, so the back projection error does not meet the adjustment conditions.

[0247] In the camera calibration method provided by the embodiments of the present disclosure, the back projection error can be used to characterize the accuracy of the internal parameters and external parameters used when calculating the coordinates. The smaller the back projection error, the higher the accuracy of the internal parameters and external parameters. Therefore, a preset error threshold is set, and the back projection error is compared with the preset error threshold. When the back projection error is large, that is, the back projection error is greater than or equal to the preset error threshold, the calibration object is adjusted until the back projection error is less than the preset error threshold. At this time, the adjustment is completed and the calibration is completed.

[0248] Therefore, the camera calibration method improves the problem of large back-projection error caused by objective factors of the calibration object itself, thereby improving the problem of low calibration accuracy of camera internal and external parameters caused by objective factors of the calibration object itself; thereby improving the calibration accuracy of the camera's external parameters, which is beneficial to reducing the stitching error of image stitching, and further beneficial to improving the image stitching effect.

[0249] The camera calibration method provided by the embodiments of the present disclosure provides a method for improving the problem of poor image stitching due to the low accuracy of the calibration external parameters by using a quantitative analysis method, namely, comparing the back-projection error with a preset error threshold. In this method, the intrinsic and extrinsic parameters can be calculated using the corner points of a partition (locally, the same chessboard grid can be considered as a partition), and then back-projected to further obtain the back-projection error. By changing the objective factors of the calibration object itself that affect the calibration, the back-projection error is reduced until the calibration requirements are met, thereby improving the calibration accuracy and the surround stitching effect.

[0250] The embodiments of the present disclosure further provide a camera calibration device, which can execute the steps of any of the above-mentioned camera calibration methods provided in the embodiments of the present disclosure to achieve corresponding beneficial effects. The similarities can be understood by referring to the explanation of the camera calibration method above, and will not be repeated here.

[0251] In the following, combined Figure 3 The camera calibration device provided in the embodiment of the present disclosure is exemplarily described.

[0252] Figure 3 A schematic structural diagram of a camera calibration device provided by an embodiment of the present disclosure is shown.

[0253] like Figure 3 As shown, the camera calibration device 30 may include:

[0254] An image acquisition module 310 is configured to acquire a first image captured by a camera to be calibrated; wherein the first image includes a reference calibration pattern;

[0255] A first determination module 320 is configured to determine external parameters of the camera to be calibrated based on the first image;

[0256] A second determination module 330 is configured to back-project the reference points in the reference calibration pattern based on the external parameters to determine a back-projection error;

[0257] The parameter adjustment module 340 is configured to adjust the target parameters of the reference calibration pattern when an adjustment condition is determined to be satisfied based on the back projection error, and return to obtain a second image of the reference calibration pattern captured by the camera to be calibrated, including the reference calibration pattern after the target parameter adjustment. Calibration is completed when the back projection error determined based on the second image no longer satisfies the adjustment condition.

[0258] In the camera calibration device 30 provided in the embodiment of the present disclosure, through the collaboration between the above-mentioned functional modules, it is possible to adjust the target parameters of the reference calibration pattern and loop the calibration process, so that when the back projection error does not meet the adjustment conditions, the calibration accuracy of the camera's external parameters is higher, thereby improving the problem of large back projection error caused by objective factors of the reference calibration pattern itself (the reference calibration pattern can be set on the calibration object, for example), thereby improving the problem of low calibration accuracy of the camera's external parameters caused by objective factors of the calibration object itself; thereby improving the calibration accuracy of the camera's external parameters, which is beneficial to reducing the stitching error of image stitching, and further beneficial to improving the image stitching effect.

[0259] In some embodiments, the camera to be calibrated has known intrinsic parameters and distortion parameters;

[0260] The first determination module 320 is configured to determine the external parameters of the camera to be calibrated based on the first image, which may specifically include:

[0261] determining coordinates of reference points in a reference calibration pattern based on the first image;

[0262] Get the actual three-dimensional coordinates corresponding to the reference point coordinates;

[0263] Determine the extrinsic parameters of the camera to be calibrated based on the reference point coordinates, the actual 3D coordinates, the intrinsic parameters, and the distortion parameters;

[0264] The second determining module 330 is configured to perform back-projection on the reference points in the reference calibration pattern based on the external parameters to determine the back-projection error, which may specifically include:

[0265] Based on the actual three-dimensional coordinates, internal parameters, distortion parameters and external parameters, back-projection is performed to obtain calculated coordinates corresponding to the coordinates of the reference point;

[0266] Based on the reference point coordinates and the corresponding calculated coordinates, the back-projection error is determined.

[0267] In some embodiments, the number of reference points is at least one, the reference point coordinates are (x, y), and the calculated coordinates are (X, Y);

[0268] The second determining module 330 is configured to determine the back-projection error based on the reference point coordinates and the corresponding calculated coordinates, and may specifically include:

[0269] The back-projection error is calculated using the following formula:

[0270]

[0271] Wherein, L represents the back-projection error, n represents the number of reference points, i represents the i-th reference point, n ≥ 1 and n is an integer, and i takes an integer value from 1 to n.

[0272] In some embodiments, the number of cameras to be calibrated is at least one, and the number of reference calibration patterns is at least one;

[0273] The second determining module 330 is configured to determine a back-projection error based on the reference point coordinates and the corresponding calculated coordinates, and may specifically include:

[0274] When the number of reference calibration patterns collected by each camera to be calibrated is j, the back-projection error is determined based on the coordinates of the reference points in the m reference calibration patterns and the corresponding calculated coordinates;

[0275] Wherein, j≥1 and j is an integer, and m takes integer values from 1 to j in sequence.

[0276] In some embodiments, the reference calibration pattern comprises a checkerboard, and the reference point in the reference calibration pattern comprises a corner point of at least one checkerboard grid;

[0277] The reference calibration pattern is set on the calibration object; correspondingly, the calibration object includes a checkerboard cloth with checkerboard patterns or a ground with checkerboard patterns;

[0278] The parameter adjustment module 340 is used to adjust the target parameters of the reference calibration pattern, which may include:

[0279] Adjust the calibration object.

[0280] In some embodiments, the calibration object includes M checkerboard grids surrounding the target object; M≥N≥3, and M and N are both integers;

[0281] Among them, N chessboard squares are respectively located at the vertices of the N corners of the N-gon surrounding the target object, and the remaining MN chessboard squares are respectively located on the N sides of the N-gon surrounding the target object;

[0282] wherein the first image includes at least one checkerboard;

[0283] The second image includes at least one checkerboard.

[0284] In some embodiments, the first determining module 320 is configured to obtain the actual three-dimensional coordinates corresponding to the reference point coordinates, which may specifically include:

[0285] Establish a world coordinate system with the projection center of the target object on the ground as the origin of the world coordinate system;

[0286] Determine the actual three-dimensional coordinates of the corner points of the chessboard in the world coordinate system;

[0287] The first determination module 320 is used to determine the external parameters of the camera to be calibrated based on the reference point coordinates, the actual three-dimensional coordinates, the internal parameters, and the distortion parameters, which may specifically include:

[0288] Based on the internal parameters and distortion parameters, the reference point coordinates are corrected for distortion;

[0289] Based on the actual 3D coordinates and the coordinates of the reference point after distortion correction, the extrinsic parameters of the camera to be calibrated are obtained.

[0290] In some embodiments, the parameter adjustment module 340 is used to adjust the calibration object, which may specifically include:

[0291] When calibrating a corresponding single camera to be calibrated using a single checkerboard, at least one of adjusting a size error of the checkerboard, adjusting the flatness of the ground, adjusting the flatness of the checkerboard cloth, and correcting an inclination of the checkerboard is adjusted;

[0292] and / or,

[0293] When calibrating a corresponding single camera to be calibrated using two checkerboards, at least one of adjusting the relative positions between the checkerboards, adjusting the spacing between the checkerboards, adjusting the flatness of the ground, and adjusting the flatness of the checkerboard cloth is adjusted;

[0294] and / or,

[0295] When calibrating a corresponding single camera to be calibrated using at least three checkerboard squares, the site is changed to adjust the ground surface, and / or the chessboard cloth is adjusted.

[0296] In some embodiments, the number of cameras to be calibrated is N;

[0297] When each camera to be calibrated obtains at least three consecutive checkerboard squares located on a single side of the N-gon, the apparatus further includes:

[0298] an auxiliary judgment module, configured to determine a back-projection error based on a single checkerboard grid, two checkerboard grids, and at least three checkerboard grids, and to judge whether an adjustment condition is satisfied based on each back-projection error;

[0299] The auxiliary adjustment module is used to adjust the calibration object when any back-projection error meets the adjustment condition, and the calibration is completed when all back-projection errors do not meet the adjustment condition.

[0300] In some embodiments, the parameter adjustment module 340 is configured to determine whether an adjustment condition is satisfied based on the back-projection error, which may specifically include:

[0301] Determining whether the back-projection error is greater than or equal to a preset error threshold;

[0302] When the back-projection error is greater than or equal to the preset error threshold, it is determined that the back-projection error meets the adjustment condition.

[0303] In some embodiments, the parameter adjustment module 340 is further configured to:

[0304] When the back-projection error is less than the preset error threshold, it is determined that the back-projection error does not meet the adjustment condition.

[0305] It should be noted that Figure 3 The camera calibration device 30 shown can perform Figure 1 The various steps in the method embodiment shown are implemented Figure 1 The various processes and effects in the illustrated method embodiment are not described in detail here.

[0306] Based on the above-mentioned embodiments, the embodiments of the present disclosure also provide an image stitching method, which includes any one of the camera calibration methods provided in the above-mentioned embodiments. On the basis of being able to improve the calibration accuracy of the camera's external parameters, it can reduce the stitching error of image stitching, thereby improving the image stitching effect.

[0307] In the following, combined Figure 4 The image stitching method provided in the embodiment of the present disclosure is exemplified.

[0308] Figure 4 A flow chart of an image stitching method provided by an embodiment of the present disclosure is shown.

[0309] like Figure 4 As shown, the image stitching method may include the following steps:

[0310] S401: Acquire images to be stitched using at least two cameras.

[0311] In this step, at least two cameras may be used to perform real-time photography or video capture, and the images captured or captured are the images to be stitched.

[0312] S402: Calibrate at least two cameras using a camera calibration method.

[0313] The camera calibration method is any one of the camera calibration methods in the above embodiments.

[0314] S403: stitching the images to be stitched based on the camera parameters determined by calibration.

[0315] In this step, registration between video images is achieved to achieve the purpose of panoramic stitching.

[0316] In the image stitching method provided by the embodiments of the present disclosure, due to the adoption of any one of the camera calibration methods in the above-mentioned embodiments, the problem of large back-projection error caused by objective factors of the calibration object itself is improved, thereby improving the problem of low calibration accuracy of camera internal and external parameters caused by objective factors of the calibration object itself; thereby improving the calibration accuracy of the camera's external parameters, which can reduce the stitching error of image stitching and improve the image stitching effect.

[0317] On the basis of the above-mentioned embodiments, the embodiments of the present disclosure further provide an image stitching device, which can execute the steps of any image stitching method provided in the embodiments of the present disclosure to achieve corresponding beneficial effects. The similarities can be understood by referring to the explanation of the image stitching method above, and will not be repeated here.

[0318] In the following, combined Figure 5 The image stitching device provided in the embodiment of the present disclosure is exemplarily described.

[0319] Figure 5 A structural schematic diagram of an image stitching device provided by an embodiment of the present disclosure is shown.

[0320] like Figure 5 As shown, the image stitching device 50 may include:

[0321] The image acquisition module 510 is configured to acquire the image to be stitched using at least two cameras;

[0322] An image calibration module 520 is configured to calibrate at least two cameras using any of the above-mentioned camera calibration methods;

[0323] The image stitching module 530 is configured to stitch the images to be stitched based on the camera parameters determined by calibration.

[0324] In the image stitching device 50 provided in the embodiment of the present disclosure, since its functional modules can implement any of the camera calibration methods in the above-mentioned embodiments, the problem of large back-projection error caused by objective factors of the calibration object itself is improved, thereby improving the problem of low calibration accuracy of camera internal parameters and external parameters caused by objective factors of the calibration object itself; thereby improving the calibration accuracy of the camera's external parameters, which can reduce the stitching error of image stitching and improve the image stitching effect.

[0325] The disclosed embodiments further provide a surround-view camera that utilizes any of the aforementioned camera calibration methods to calibrate at least two cameras, or utilizes any of the aforementioned image stitching methods to stitch surround-view images. Consequently, the surround-view camera's external parameters are calibrated with high accuracy, resulting in smaller stitching errors during image stitching and better image stitching results.

[0326] The disclosed embodiments further provide a vehicle comprising any of the aforementioned surround-view cameras. The surround-view camera can be used in an on-vehicle panoramic view system. The camera's external parameters are calibrated with high accuracy, resulting in low stitching errors during image stitching, thus achieving a good image stitching effect. This enables the on-vehicle panoramic view system to more accurately identify the environment surrounding the vehicle, providing more accurate image information for autonomous or assisted driving.

[0327] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a..." does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0328] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A camera calibration method, characterized in that: include: Acquire a first image captured by the camera to be calibrated; wherein the first image includes a reference calibration pattern; Determining external parameters of the camera to be calibrated based on the first image; Back-projecting reference points in the reference calibration pattern based on the extrinsic parameters to determine a back-projection error; When it is determined based on the back projection error that an adjustment condition is satisfied, the target parameters of the reference calibration pattern are adjusted, and a second image of the reference calibration pattern including the adjusted target parameters, acquired by the camera to be calibrated, is returned to be acquired, until the back projection error determined based on the second image no longer satisfies the adjustment condition, and the calibration is completed; The camera to be calibrated has known internal parameters and distortion parameters; The step of determining the external parameters of the camera to be calibrated based on the first image includes: determining coordinates of reference points in the reference calibration pattern based on the first image; Obtaining the actual three-dimensional coordinates corresponding to the reference point coordinates; Determining the extrinsic parameters of the camera to be calibrated based on the reference point coordinates, the actual three-dimensional coordinates, the intrinsic parameters, and the distortion parameters; The back-projecting of the reference points in the reference calibration pattern based on the external parameters to determine the back-projection error includes: Back-projecting to obtain calculated coordinates corresponding to the reference point coordinates based on the actual three-dimensional coordinates, the internal parameters, the distortion parameters, and the external parameters; Determining a back-projection error based on the reference point coordinates and the corresponding calculated coordinates; The number of the reference point is at least one, the coordinates of the reference point are (x, y), and the calculated coordinates are (X, Y); The step of determining the back-projection error based on the reference point coordinates and the corresponding calculated coordinates includes: The back-projection error is calculated using the following formula: Wherein, L represents the back-projection error, n represents the number of reference points, i represents the i-th reference point, n ≥ 1 and n is an integer, and i takes an integer value from 1 to n.

2. The method according to claim 1, characterized in that The number of the cameras to be calibrated is at least one, and the number of the reference calibration patterns is at least one; When the number of the reference calibration patterns collected by each camera to be calibrated is j, determining the back projection error based on the reference point coordinates and the corresponding calculated coordinates includes: determining a back-projection error based on the coordinates of the reference points in the m reference calibration patterns and the corresponding calculated coordinates; Wherein, j≥1 and j is an integer, and m takes integer values from 1 to j in sequence.

3. The method according to any one of claims 1-2, characterized in that The reference calibration pattern includes a checkerboard, and the reference point in the reference calibration pattern includes at least one corner point of the checkerboard; The reference calibration pattern is set on a calibration object; correspondingly, the calibration object includes a checkerboard cloth with checkerboard patterns or a ground with checkerboard patterns; Wherein, adjusting the target parameters of the reference calibration pattern includes: The calibration object is adjusted.

4. The method according to claim 3, characterized in that The calibration object includes M checkerboard grids surrounding the target object; M≥N≥3, and M and N are both integers; Wherein, N checkerboard squares are respectively located at the vertices of the N corners of the N-gon surrounding the target object, and the remaining MN checkerboard squares are respectively located on the N sides of the N-gon surrounding the target object; wherein the first image includes at least one checkerboard; The second image includes at least one checkerboard.

5. The method according to claim 3, characterized in that The obtaining of the actual three-dimensional coordinates corresponding to the reference point coordinates includes: Establish a world coordinate system with the projection center of the target object on the ground as the origin of the world coordinate system; Determining the actual three-dimensional coordinates of the corner points of the chessboard in the world coordinate system; The determining of the external parameters of the camera to be calibrated based on the reference point coordinates, the actual three-dimensional coordinates, the internal parameters, and the distortion parameters includes: Performing distortion correction on the reference point coordinates based on the internal parameters and the distortion parameters; Based on the actual three-dimensional coordinates and the distortion-corrected coordinates of the reference point, the external parameters of the camera to be calibrated are obtained.

6. The method according to claim 3, characterized in that The adjusting the calibration object includes: When calibrating a corresponding single camera to be calibrated using a single checkerboard, at least one of adjusting a size error of the checkerboard, adjusting the flatness of the ground, adjusting the flatness of the checkerboard cloth, and correcting an inclination of the checkerboard is adjusted; and / or, When calibrating a corresponding single camera to be calibrated using two checkerboards, at least one of adjusting the relative positions between the checkerboards, adjusting the spacing between the checkerboards, adjusting the flatness of the ground, and adjusting the flatness of the checkerboard cloth; and / or, When calibrating a corresponding single camera to be calibrated using at least three checkerboard squares, the site is changed to adjust the ground, and / or the layout of the checkerboard cloth is adjusted.

7. The method according to claim 4, characterized in that The number of cameras to be calibrated is N; When each of the cameras to be calibrated respectively acquires at least three consecutive checkerboard squares located on a single side of the N-gon, the method further includes: Determining the back-projection error based on a single checkerboard grid, two checkerboard grids, and at least three checkerboard grids, respectively, and determining whether an adjustment condition is satisfied based on each back-projection error; When any of the back-projection errors satisfies the adjustment condition, the calibration object is adjusted until the calibration ends when all the back-projection errors fail to satisfy the adjustment condition.

8. The method according to claim 1, characterized in that The determining, based on the back-projection error, that an adjustment condition is satisfied includes: Determining whether the back-projection error is greater than or equal to a preset error threshold; When the back-projection error is greater than or equal to the preset error threshold, it is determined that the back-projection error meets an adjustment condition.

9. The method according to claim 8, characterized in that After determining whether the back-projection error is greater than or equal to a preset error threshold, the method further includes: When the back-projection error is less than the preset error threshold, it is determined that the back-projection error does not meet the adjustment condition.

10. A camera calibration device, characterized in that: include: An image acquisition module, configured to acquire a first image captured by a camera to be calibrated; wherein the first image includes a reference calibration pattern; A first determining module, configured to determine external parameters of the camera to be calibrated based on the first image; a second determining module, configured to back-project the reference points in the reference calibration pattern based on the external parameters to determine a back-projection error; a parameter adjustment module, configured to adjust target parameters of the reference calibration pattern when an adjustment condition is determined to be satisfied based on the back projection error, and return to obtain a second image of the reference calibration pattern acquired by the camera to be calibrated, including the reference calibration pattern after the target parameter adjustment, until the back projection error determined based on the second image no longer satisfies the adjustment condition, indicating that calibration is complete; The camera to be calibrated has known internal parameters and distortion parameters; The first determining module is used to determine the external parameters of the camera to be calibrated based on the first image, and may specifically include: determining coordinates of reference points in the reference calibration pattern based on the first image; Obtaining the actual three-dimensional coordinates corresponding to the reference point coordinates; Determining the extrinsic parameters of the camera to be calibrated based on the reference point coordinates, the actual three-dimensional coordinates, the intrinsic parameters, and the distortion parameters; The second determining module is configured to perform back-projection on the reference points in the reference calibration pattern based on the external parameters to determine the back-projection error, which may specifically include: Back-projecting to obtain calculated coordinates corresponding to the reference point coordinates based on the actual three-dimensional coordinates, the internal parameters, the distortion parameters, and the external parameters; Determining a back-projection error based on the reference point coordinates and the corresponding calculated coordinates; The number of the reference point is at least one, the coordinates of the reference point are (x, y), and the calculated coordinates are (X, Y); The second determining module is configured to determine the back-projection error based on the reference point coordinates and the corresponding calculated coordinates, and may specifically include: The back-projection error is calculated using the following formula: Wherein, L represents the back-projection error, n represents the number of reference points, i represents the i-th reference point, n ≥ 1 and n is an integer, and i takes an integer value from 1 to n.

11. An image stitching method, characterized in that: include: Acquiring images to be stitched using at least two cameras; Calibrate the at least two cameras using the camera calibration method according to any one of claims 1 to 9; The images to be stitched are stitched based on the camera parameters determined by calibration.

12. A surround view camera, characterized in that: Calibrate at least two cameras using the camera calibration method according to any one of claims 1 to 9; or The image stitching method according to claim 11 is used to stitch surround view images.

13. A vehicle, characterized in that: Including the surround-view camera as described in claim 12.

Images