Camera calibration methods, devices, electronic equipment, storage media and products

Abstract

This application provides a camera calibration method, apparatus, electronic device, storage medium, and product. The method includes: acquiring data from a target using at least one camera, and determining the grayscale image and point cloud data corresponding to each camera; the target includes multiple sub-regions divided by mutually perpendicular straight lines, a region positioning code located in each sub-region, and a center positioning code located at the center of the target; determining the target region observed by each camera based on the region positioning code identified from the grayscale image; performing plane fitting based on the point cloud data corresponding to the target region to obtain a reference plane; performing line detection on the grayscale image region corresponding to the target region, and determining the target reference line from the detected lines based on the center positioning code; and determining the extrinsic parameter data of each camera based on the intrinsic parameter data of each camera, the reference plane, and the target reference line. This application improves the efficiency and accuracy of camera extrinsic parameter calibration.

Description

Technical Field

[0001] This application relates to sensor parameter calibration technology, and more particularly to a camera calibration method, apparatus, electronic device, storage medium, and product. Background Technology

[0002] Currently, in 3D reconstruction and measurement applications using depth cameras, it is necessary to calibrate the extrinsic parameters of multiple cameras. Some related techniques, such as the Perspective-n-Point (PnP) algorithm, have strict limitations on the camera field of view and the layout of the calibration scene when calibrating the extrinsic parameters of multiple cameras simultaneously, resulting in a limited number of cameras that can be calibrated at once, thus reducing the efficiency of camera calibration. Furthermore, the PnP algorithm needs to infer 3D pose information from 2D observation information, which is prone to problems such as scale instability and amplification of depth direction errors, thereby reducing the accuracy of camera extrinsic parameter calibration. Other related techniques calibrate multiple cameras' extrinsic parameters by calibrating cameras pairwise and then stitching them together. This method is cumbersome, and errors accumulate step by step during the stitching process, reducing the accuracy of camera extrinsic parameter calibration. In summary, the efficiency and accuracy of current related techniques for camera extrinsic parameter calibration are relatively low. Summary of the Invention

[0003] This application provides a camera calibration method, apparatus, electronic device, storage medium, and product that can improve the efficiency and accuracy of camera extrinsic parameter calibration.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a camera calibration method, the method including:

[0006] Point cloud and image acquisition of the target are performed by each of at least one camera, and grayscale image and point cloud data of each camera are determined; the target includes: multiple sub-regions divided by at least two mutually perpendicular straight lines, at least one region localization code of each sub-region, and center localization code of the target center;

[0007] The grayscale image is subjected to location code recognition to determine the center location code and at least one target region location code, and the target region observed by each camera is determined based on at least one target region location code; the target region includes at least one target sub-region among multiple sub-regions;

[0008] Plane fitting is performed based on the point cloud data corresponding to the target area to determine the reference plane for each camera.

[0009] Line detection is performed on the grayscale image region corresponding to the target area, and the target reference line corresponding to each camera is determined from the detected lines based on the center localization code.

[0010] Based on the intrinsic parameter data of each camera, the reference plane, and the target reference line, the extrinsic parameter data of each camera are determined.

[0011] This application provides a camera calibration device, including: a data acquisition module, used to acquire point cloud and image data of a target through each of at least one camera, and determine the grayscale image and point cloud data of each camera; the target includes: multiple sub-regions divided by at least two mutually perpendicular straight lines, at least one region positioning code for each sub-region, and a center positioning code for the center of the target.

[0012] The calibration module is used to identify positioning codes in grayscale images, determine the center positioning code and at least one target region positioning code, and determine the target region observed by each camera based on at least one target region positioning code; the target region includes at least one target sub-region among multiple sub-regions; perform plane fitting based on the point cloud data corresponding to the target region to determine the reference plane corresponding to each camera; perform line detection on the grayscale image region corresponding to the target region, and determine the target reference line corresponding to each camera from the detected lines based on the center positioning code; and determine the extrinsic data of each camera based on the intrinsic parameter data of each camera, the reference plane, and the target reference line.

[0013] This application provides a camera calibration system, including a target, at least one camera, and a processing unit, wherein...

[0014] The target includes: multiple sub-regions divided by at least two mutually perpendicular straight lines, at least one region location code for each sub-region, and a center location code for the center of the target;

[0015] Each of at least one camera is used to acquire point cloud and image data of the target, and to determine the grayscale image and point cloud data of each camera;

[0016] The processing unit is used to perform location code recognition on the grayscale image, determine the center location code and at least one target region location code, and determine the target region observed by each camera based on the at least one target region location code; the target region includes at least one target sub-region among multiple sub-regions; perform plane fitting based on the point cloud data corresponding to the target region to determine the reference plane corresponding to each camera; perform line detection on the grayscale image region corresponding to the target region, and determine the target reference line corresponding to each camera from the detected lines based on the center location code; and determine the extrinsic data of each camera based on the intrinsic parameter data of each camera, the reference plane and the target reference line.

[0017] This application provides an electronic device, including:

[0018] Memory is used to store executable instructions or computer programs.

[0019] A processor, when executing computer-executable instructions or computer programs stored in memory, implements the methods provided in the embodiments of this application.

[0020] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the camera calibration method provided in this application when executed by a processor.

[0021] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the camera calibration method provided in this application.

[0022] The embodiments of this application have the following beneficial effects: Point cloud and image acquisition of the target are performed by each of at least one camera, determining the grayscale image and point cloud data of each camera; wherein the target includes multiple sub-regions divided by at least two mutually perpendicular straight lines, at least one region positioning code in each sub-region, and a center positioning code at the center of the target. Thus, as long as the camera can observe the center positioning code and any region positioning code on the target, the target target region observed by each camera can be determined using at least one target region positioning code observed by each camera, and the center positioning code observed by each camera provides a globally consistent position reference, accurately unifying the local geometric features fitted by each camera into the same reference coordinate system. Therefore, the limitations on camera field of view and calibration scene layout are reduced, greatly improving the flexibility of calibration scene layout and the number of cameras that can be calibrated at one time, thus improving calibration efficiency. During the calibration process, plane fitting is performed using the point cloud data corresponding to the target area observed by each camera to determine the reference plane for each camera. Line detection is then performed based on the grayscale image area corresponding to the target area, and a target reference line for each camera is determined from the detected lines based on the center localization code. Finally, based on the intrinsic parameter data of each camera, the reference plane, and the target reference line, the extrinsic parameter data for each camera is determined. This embodiment directly uses the 3D point cloud data acquired by the camera for geometric fitting, resulting in smaller errors and higher accuracy. Furthermore, the reference plane and target reference line, as high-dimensional and stable geometric features, can construct stronger geometric constraints in the camera coordinate system. This, combined with the center localization code and camera intrinsic parameters, allows for the solution of the extrinsic parameter data for each camera, thereby improving the calibration accuracy of the extrinsic parameter data. Thus, the efficiency and accuracy of camera extrinsic parameter calibration are improved. Attached Figure Description

[0023] Figure 1This is a schematic flowchart of an optional camera calibration method provided in an embodiment of this application;

[0024] Figure 2 This is a schematic diagram illustrating an optional effect of the target provided in an embodiment of this application;

[0025] Figure 3 This is a schematic diagram illustrating the effect of the target location identifier corresponding to the target area location code provided in the embodiments of this application;

[0026] Figure 4 This is a schematic diagram illustrating the effect of the grayscale image region corresponding to the target region provided in the embodiments of this application;

[0027] Figure 5 This is a schematic diagram illustrating the effect of the reference straight line provided in the embodiments of this application;

[0028] Figure 6 This is an optional flowchart illustrating a camera calibration method applied to a real-world scenario, as provided in an embodiment of this application.

[0029] Figure 7 This is a schematic diagram of the composition structure of a camera calibration device provided in an embodiment of this application;

[0030] Figure 8 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] In 3D camera reconstruction and measurement scenarios, it is necessary to calibrate the extrinsic parameters of multiple cameras. Current technologies sometimes only calibrate one camera at a time, making simultaneous calibration of multiple cameras inefficient. Other technologies, when calibrating multiple cameras, generally employ target-based camera extrinsic parameter calibration methods, including common algorithms such as PnP, stereoCalibrate, and geometrically constrained 3D-to-3D calibration algorithms.

[0033] Among them, the PnP algorithm is the general term for the classic problem in computer vision: "Given n 3D world points and their projections onto a 2D image, find the pose (R,t) of the camera in the world coordinate system." PnP calibration requires inferring 3D pose information from 2D observation information, relying on the observation of known 3D points by a single camera. The extrinsic parameters are strongly affected by the accuracy of corner detection, depth, or scale priors, which can easily lead to problems such as scale instability and amplification of depth direction errors. When the chessboard pose variation is insufficient or the point distribution is degraded, the solution may not be unique, and the extrinsic parameters may appear stable but lack physical realism.

[0034] The stereoCalibrate algorithm is based on binocular geometric constraints. Although it can estimate the relative pose of two cameras simultaneously, it is highly sensitive to synchronization, target coverage of the field of view, baseline (the length of the line connecting the optical centers of the left and right cameras of the binocular camera) and parallax. A small baseline or insufficient parallax will lead to a decrease in depth and scale accuracy. In addition, its results are relative extrinsic parameters. If a single target is not accurately located or corner errors accumulate, the final physical position and attitude may still have systematic deviations.

[0035] The geometrically constrained 3D-to-3D calibration method uses the 3D coordinates of multiple sets of corresponding points to solve for extrinsic parameters by minimizing the geometric error under rigid body transformation. However, this method is highly susceptible to errors in the coordinates of a single point, which reduces the accuracy of extrinsic parameter calibration.

[0036] This application provides a camera calibration method, apparatus, electronic device, storage medium, and product that can calibrate multiple camera extrinsic parameters based on point cloud data and grayscale images acquired by the camera. It has the advantages of being simple, efficient, highly accurate, and able to handle partial occlusion or small field of view when cameras overlap.

[0037] See Figure 1 , Figure 1 This is a schematic flowchart of an optional camera calibration method provided in an embodiment of this application, including:

[0038] S101. The target is acquired by point cloud and image acquisition by each of at least one camera, and the grayscale image and point cloud data of each camera are determined.

[0039] This application embodiment is applicable to calibration scenarios where the extrinsic parameter data of each camera in at least one camera is determined through a single calibration. In this application embodiment, the extrinsic parameter data describes the pose of the camera in a reference coordinate system and is used to establish a rigid body transformation relationship from the reference coordinate system to the camera coordinate system. The reference coordinate system in this application embodiment refers to a fixed world coordinate system established with the target as a reference, which can also be called an absolute coordinate system.

[0040] In this embodiment, at least one camera is positioned within the same calibration scene, performing point cloud and image acquisition on the same target from at least one perspective, resulting in grayscale images and point cloud data for each camera. The acquisition perspectives of different cameras may be the same or different; no specific limitation is made here.

[0041] The camera in this embodiment has the function of acquiring depth data (such as three-dimensional point cloud data) and intensity data (two-dimensional image data). Exemplarily, the camera may include, but is not limited to, structured light depth cameras, binocular stereo vision cameras, and time-of-flight (TOF) cameras, etc., selected according to actual conditions; this embodiment does not limit the choice. Taking a TOF camera as an example, in the same calibration scene, at least one TOF camera can acquire point cloud data and images of the same target. For each TOF camera, aligned point cloud data and grayscale images are obtained synchronously. The point cloud data and grayscale images can be used to provide the spatial correspondence between three-dimensional coordinate points and two-dimensional coordinate points. In some embodiments, the grayscale image may include an infrared image acquired by the TOF camera, and the point cloud data may be converted from a depth map acquired by the TOF camera; the specific selection depends on actual conditions, and this embodiment does not limit the choice.

[0042] The target in this embodiment is a continuous plane, comprising: multiple sub-regions divided by at least two mutually perpendicular straight lines, at least one region positioning code for each sub-region, and a center positioning code for the target center. Each sub-region has at least one region positioning code, allowing the camera to locate a local plane of the target based on one or more region positioning codes without needing to observe the entire target. This allows for geometric fitting using the point cloud data of the local plane. In other words, this embodiment places fewer restrictions on the camera's field of view and the setup of the calibration scene when calibrating multiple cameras. Even with a small overlap in the field of view of multiple cameras, as long as the center positioning code and at least one region positioning code are observed simultaneously, the mapping relationship between the camera and the global target coordinate system can be determined based on the observed local plane of the target, local line segments, and positioning codes, thus achieving extrinsic parameter calibration of the camera.

[0043] As can be seen, the embodiments of this application do not require the camera to see the entire target, reducing the stringent requirements for camera layout and improving robustness. Cameras participating in the calibration only need to see the center positioning code and at least one area positioning code to achieve calibration, resulting in higher scalability. Furthermore, by placing the target once, all cameras whose fields of view cover the target (partially or entirely) can be calibrated simultaneously without moving the target or calibrating them one by one, thus improving efficiency. Therefore, highly scalable, highly robust, and highly efficient parallel calibration is achieved.

[0044] The location code in this application embodiment can be implemented using any coded pattern that has a unique identifier and can be reliably detected in the image. For example, it may include ArUco (Augmented Reality University of Cordoba marker) code, AprilTag visual reference tag, Quick Response (QR) code, checkerboard corner dot array, or custom binary coded mark, etc. The specific selection is based on the actual situation, and this application embodiment does not limit it.

[0045] S102. Perform positioning code recognition on the grayscale image, determine the center positioning code and at least one target area positioning code, and determine the target area observed by each camera based on at least one target area positioning code.

[0046] In this embodiment of the application, the grayscale image acquired by each camera is subjected to positioning code recognition to determine the center positioning code observed by the camera, and at least one area positioning code observed by the camera is identified as at least one target area positioning code.

[0047] In this embodiment, the target region includes at least one target sub-region among multiple sub-regions. Based on at least one target region positioning code, the target plane portion observed by each camera, i.e., the target region, can be determined. The target region includes one or more sub-regions among multiple sub-regions on the target observed by the camera, serving as at least one target sub-region.

[0048] S103. Perform plane fitting based on the point cloud data corresponding to the target area to determine the reference plane for each camera.

[0049] In this embodiment, based on the target region, the point cloud data corresponding to the target region in the point cloud data acquired by the camera can be determined. Plane fitting is then performed using the point cloud data corresponding to the target region. For example, the coordinates of the three-dimensional points in the point cloud data corresponding to the target region can be used to perform plane fitting using algorithms such as Random Sample Consensus (RANSAC), least squares, and Principal Component Analysis (PCA) to obtain a reference plane for each camera. The specific choice depends on the actual situation, and this embodiment does not limit the choice.

[0050] In some embodiments, the parameters in the three-dimensional plane equations can be solved by fitting the known three-dimensional plane equations based on the point cloud data corresponding to the target region, thereby determining the reference plane for each camera. The parameters of the three-dimensional plane equations for each camera represent the local geometric expression of the real physical plane on the target from the camera's viewpoint in the camera coordinate system. The reference plane can provide spatial orientation and positional constraints of the target plane in the camera coordinate system.

[0051] As can be seen, the embodiments of this application directly use the 3D point cloud acquired by the camera for plane fitting, and the scale is known and uniform. In contrast, methods based on pure image point correspondence (such as PnP) in related technologies indirectly calculate depth information through geometric relationships, resulting in scale ambiguity. Furthermore, errors in the depth direction are significantly amplified by the perspective projection model, leading to instability in the calibration results in terms of scale or depth. Therefore, compared to related technologies, the embodiments of this application provide stricter geometric constraints through the reference plane.

[0052] It should be noted that in this embodiment, each of the multiple cameras participating in a calibration is independently fitted to a reference plane. That is, each camera independently fits a reference plane based on its own observed point cloud data. If the observation viewpoints (i.e., their respective camera coordinate systems) of the cameras are different, the three-dimensional plane equations of the reference planes they fit will also differ in parameters. However, it should be understood that these different plane equations physically describe the same calibration plane on the target.

[0053] S104. Perform line detection on the grayscale image region corresponding to the target region, and determine the target reference line corresponding to each camera from the detected lines based on the center positioning code.

[0054] In this embodiment, a straight line is detected in the grayscale image region corresponding to the target region in the grayscale image. Then, the target center corresponding to the position of the center positioning code is used as the origin of the reference coordinate system. The straight line with the highest correlation to the origin is determined from the detected straight lines and used as the target reference straight line for each camera.

[0055] For example, the line closest to the origin represented by the position of the center positioning code can be determined from the detected lines and used as the target reference line.

[0056] As can be seen, by performing line detection and filtering on the two-dimensional grayscale image region corresponding to the target area, the target reference line corresponding to each camera is determined, providing a precise directional reference for establishing a unified reference coordinate system. It can be understood that, since the target is a continuous plane, the target reference line lies on the reference plane obtained above through fitting.

[0057] S105. Based on the intrinsic parameter data of each camera, the reference plane, and the target reference line, determine the extrinsic parameter data of each camera.

[0058] In this embodiment, the intrinsic parameter data of each camera can be obtained through a pre-defined single-camera calibration process. The intrinsic parameter data describes the camera's inherent internal geometric and optical characteristics, which do not change with the shooting scene or position. It is used to establish a mapping relationship between three-dimensional points in the camera coordinate system and two-dimensional projection points in its image coordinate system. For example, the intrinsic parameter data may include an intrinsic parameter matrix, which defines inherent parameters such as the camera's equivalent focal length (f_x, f_y) and principal point (c_x, c_y). This matrix can project three-dimensional points from the point cloud data into the two-dimensional image coordinate system (the coordinate system where the grayscale image is located) in the camera coordinate system.

[0059] In this embodiment, extrinsic parameter data is used for coordinate transformation between the camera coordinate system of each camera and the reference coordinate system defined by the target. For example, the extrinsic parameter data may include an extrinsic parameter matrix.

[0060] In this embodiment, the local geometric features (i.e., the reference plane and the target reference line) independently observed and extracted by each camera are matched and aligned with the imaging model established by the camera intrinsic data and the globally unified geometric model known by the target itself with high precision, thereby determining the extrinsic data of each camera.

[0061] In this embodiment, a reference plane can provide the spatial orientation and positional constraints of the target plane in the camera coordinate system; a target reference line can provide the absolute orientation reference within the target plane. In some embodiments, the X-axis or Y-axis of the reference coordinate system can be determined based on the direction of the target reference line for each camera; the origin of the reference coordinate system can be determined based on the position information of the center positioning code; and the Z-axis of the reference coordinate system can be determined by combining the spatial orientation of the target plane in the camera coordinate system determined by the reference plane. This constitutes the geometric constraint basis required for a complete and unambiguous transformation from the camera coordinate system to the reference coordinate system. The target reference line can be represented using mathematical models such as parametric linear equations, and the reference plane can be represented using mathematical models such as parametric plane equations.

[0062] In this embodiment, the camera's intrinsic parameter data accurately describes the camera's imaging geometry. By using the intrinsic parameter data, the parameters of the aforementioned target reference line and reference plane are uniformly correlated in three-dimensional space for mathematical solution. An optimization algorithm is then used to solve for the transformation parameters that satisfy these strong geometric constraints, thereby obtaining high-precision and high-stability camera extrinsic parameter data.

[0063] In some embodiments, an optimization algorithm can be used to find an optimal transformation (i.e., extrinsic parameter data) such that the observed features in the camera coordinate system (defined by the reference plane and the target reference line) can be optimally aligned with the reference coordinate system after being transformed to the reference coordinate system defined by the target. The extrinsic parameter data can then be determined based on this optimal transformation. It can be seen that, compared to current calibration methods based on discrete points (such as PnP), this embodiment utilizes the "reference plane" and "target reference line" as precise "surface" and "line" constraints, providing higher-dimensional and more robust geometric constraints, thereby enabling the determination of more accurate extrinsic parameter data.

[0064] For example, extrinsic parameter data can be obtained by constructing reprojection error or geometric error equations and solving them through least squares, nonlinear optimization, etc. The specific optimization algorithm is selected according to the actual situation, and the embodiments of this application are not limited.

[0065] It is understood that, in this embodiment of the application, as long as the camera can observe the center positioning code and any area positioning code on the target, the calibration of the camera can be completed. This greatly reduces the limitations on the camera's field of view and the layout of the calibration scene when calibrating multiple cameras, significantly improves the flexibility of the calibration scene layout and the number of cameras that can be calibrated at one time, and improves the calibration efficiency. Furthermore, in the plane fitting process, the three-dimensional point cloud data collected by the camera is directly used for geometric fitting, resulting in smaller errors and higher accuracy. In this way, the reference plane and the target reference line, as high-dimensional and stable geometric features, can construct stronger geometric constraints (surface and line constraints) in the camera coordinate system. Then, the extrinsic parameter data of each camera can be solved by combining the center positioning code and the camera's intrinsic parameters, thereby improving the calibration accuracy of the extrinsic parameter data. Thus, the efficiency and accuracy of camera extrinsic parameter calibration are improved.

[0066] In some embodiments, the target is placed in the overlapping field of view of multiple cameras such that each camera can observe at least a center positioning code and at least one area positioning code; the intersection of at least two mutually perpendicular straight lines falls on the center positioning code; at least one area positioning code for each sub-region is distributed circumferentially on the target in each sub-region; each area positioning code corresponds to a unique positioning identifier.

[0067] Here, the intersection of at least two mutually perpendicular straight lines falls on the center localization code, thus strongly correlating the intersection of the straight line features with the position of the center localization code. This ensures that the origin of the target's reference coordinate system (defined by the center localization code) and the geometric center of the orientation reference (defined by the intersection of mutually perpendicular straight lines) coincide or are close in space. This provides a stable, unique, and unambiguous reference coordinate system anchor point for the calibration algorithm. At least one regional localization code for each sub-region is distributed along the circumference of the target in each sub-region, further reducing the limitation on the size of the overlapping field of view of multiple cameras. Each regional localization code corresponds to a unique localization identifier. Thus, the target region observed by the camera can be determined based on the localization identifier identified from the grayscale image and the known layout of regional localization codes on the target.

[0068] For example, the target provided in the embodiments of this application can be as follows: Figure 2 As shown, Figure 2 In the target plane, at least two mutually perpendicular lines are included: lines 20, 21, 22, and 23. Lines 20 and 22 are perpendicular; lines 20 and 23 are perpendicular; lines 21 and 22 are perpendicular; and lines 21 and 23 are perpendicular. The areas between the lines—namely, between lines 20 and 21, and between lines 22 and 23—are black, while the areas between the lines and the target boundary are white. Thus, lines 20, 21, 22, and 23 divide the target plane into four white sub-regions: sub-region 24, sub-region 25, sub-region 26, and sub-region 27. It should be noted that other high-contrast colors can also be used to distinguish the line regions and sub-regions. Figure 2 The black and white colors are merely examples. A center positioning code 28 is located at the center of the target, and area positioning codes (e.g., [missing information]) are located along the circumference of each sub-region. Figure 2 (As shown in 29_1 to 29_16). Each area location code and the center location code are ArUco codes with different patterns. Here, Figure 2 The number, arrangement, and pattern of the location codes shown are only examples. In actual applications, the number of location codes for each sub-region, the interval between location codes, and the specific pattern can be set according to the actual situation. The specific selection is based on the actual situation and is not limited in this application embodiment.

[0069] In some embodiments, the grayscale image of the target captured by the camera is subjected to location code recognition, and the corner coordinates of the center location code are output. For example, the coordinates of the upper left corner of the center location code can be output; these corner coordinates will serve as a geometric reference for subsequent filtering of line detection results; and at least one target location identifier corresponding to a target region location code is output, thereby determining the target region observed by the camera based on a preset correspondence between the location identifier and the sub-region. For example, each region location code is pre-assigned an ID as its unique location identifier, using the grayscale image captured by a certain camera as... Figure 2 For example, through Figure 2 By recognizing the pattern of the central area positioning code, the target positioning identifier (id) corresponding to the target area positioning code can be identified, such as... Figure 3 As shown, the target area observed by the camera can be determined based on the identified target location markers and the known distribution of each location marker on the target. It can be understood that even if the camera only observes part of the target, the observed target area can be determined by at least one location marker of at least one identified target area location code.

[0070] It is understood that the target in this embodiment is divided into "grid" sections, so that each sub-region (equivalent to the four quadrants of the grid) has similar local features (coplanar planar sub-regions and some straight lines). A unique regional positioning code can distinguish the target regions observed by each camera, and a central positioning code provides the origin of the reference coordinate system. Thus, even if the camera can only see a portion of the target, it can determine the position of the seen portion within the entire target using the identified positioning code, thereby completing calibration using local features and improving the efficiency and accuracy of extrinsic parameter calibration.

[0071] It should be noted that, Figure 2 The target shown is only an example of one implementation. In some embodiments, at least two mutually perpendicular lines may also include two or three mutually perpendicular lines 20, 21, 22 and 23, which are selected according to the actual situation. This application does not limit the specific implementation.

[0072] In some embodiments, the process of determining the reference plane for each camera by performing plane fitting based on the point cloud data corresponding to the target region in S103 above includes:

[0073] At least three points are selected from the point cloud data corresponding to the target area for multiple plane fittings to determine multiple candidate reference planes; a reference plane is determined from the multiple candidate reference planes based on the average distance from the points in the target area to each candidate reference plane.

[0074] In this embodiment, based on the target region observed by each camera, corresponding point cloud data is determined from the point cloud data collected by each camera. From the point cloud data corresponding to the target region, at least three points are selected each time, and plane fitting is performed using these at least three points. The plane obtained from the plane fitting is determined as a candidate reference plane. At least three points must not be collinear. Multiple candidate reference planes are determined through multiple plane fitting processes described above.

[0075] For example, the Random Sample Consensus (RANSAC) algorithm can be used for plane fitting. RANSAC can robustly estimate model parameters even in the presence of a large number of outliers through random sampling and consistency verification. For each plane fitting, three non-collinear points are randomly selected from the point cloud data corresponding to the target region. Based on these three non-collinear points and the known expression of the plane equation, the parameters in the plane equation are calculated, thus obtaining a plane equation with known parameters, representing the fitted candidate reference plane.

[0076] For example, the plane defined by points P1, P2, and P3 can be represented by the plane normal vector as follows: The normal vector of this plane can usually be normalized to make it... The normalized unit normal vector is obtained. Normalized Substitute the coordinates of any point among P1, P2, and P3 into the plane equation. This yields the value of d in the plane equation. For example, consider... and the coordinates of point P1 Substituting into the above plane equation, we can obtain Thus, an expression for the equation of a plane with known coordinates a, b, c, and d is determined. The plane represented is equivalent to obtaining a candidate reference plane through fitting. Three non-collinear points are then randomly selected from the point cloud data corresponding to the target region, and the process is repeated for the next iteration to determine another candidate reference plane, thus allowing for the determination of multiple candidate reference planes.

[0077] In some embodiments, when a preset number of iterations is reached, the iteration is stopped, and a reference plane is selected from the multiple candidate reference planes obtained by fitting.

[0078] In this embodiment, based on the points in the target region (i.e., some or all of the points in the point cloud data corresponding to the target region), the average distance from these points to one of the multiple candidate reference planes is calculated, thereby obtaining the average distance from the points in the target region to each candidate reference plane. Based on the average distance from the points in the target region to each candidate reference plane, a candidate reference plane that meets the requirements is determined from the multiple candidate reference planes and used as the reference plane. For example, the candidate reference plane with the smallest average distance can be determined as the reference plane. That is, the average distance from the points in the target region to the reference plane is minimized.

[0079] In some embodiments, after determining a plurality of candidate reference planes, the method further includes:

[0080] Determine multiple distance values ​​from multiple points in the target area to each candidate reference plane; update the distance values ​​that are greater than a preset distance threshold to the preset distance threshold, and determine the average distance based on the updated multiple distance values.

[0081] In this embodiment, when iteration stops and multiple candidate reference planes are determined, multiple distance values ​​from multiple points in the target region to each candidate reference plane are determined, and the average distance from the multiple points in the target region to each candidate reference plane is further calculated based on the multiple distance values. Alternatively, when a candidate reference plane is obtained each time through fitting, multiple distance values ​​from the multiple points in the target region to that candidate reference plane are determined, and the average distance from the multiple points in the target region to that candidate reference plane is further calculated based on the multiple distance values. Here, the multiple points in the target region include some or all of the points in the point cloud data corresponding to the target region.

[0082] Specifically, for a candidate reference plane, among multiple distance values ​​from multiple points in the target region to the candidate reference plane, distance values ​​greater than a preset distance threshold are updated to the preset distance threshold, while distance values ​​less than or equal to the preset distance threshold remain unchanged, thus obtaining updated distance values. The average distance from the multiple points to the candidate reference plane is calculated based on the average of these updated distance values.

[0083] In some embodiments, the above-mentioned multiple distance values ​​are updated and the average distance is calculated for each of the multiple candidate reference planes to obtain the average distance from multiple points in the target area to each candidate reference plane, and then the candidate reference plane with the smallest average distance is determined as the reference plane.

[0084] Understandably, in current related technologies, the RANSAC algorithm typically calculates the distances from multiple points in the point cloud to the fitted plane, then designates points with distances less than or equal to a preset distance threshold as inliers and points with distances greater than the preset distance threshold as outliers, selecting the plane with the most inliers from the fitted planes. Compared to current related technologies, this application's embodiment updates the distance values ​​greater than the preset distance threshold to the preset distance threshold, determines the average distance based on the updated distance values, and selects a reference plane based on the average distance. This allows for the acquisition of a more stable plane even in the presence of significant noise or missing data, thereby improving the robustness of the calibration algorithm.

[0085] In some embodiments, the process of performing line detection on the grayscale image region corresponding to the target region and determining the target reference line corresponding to each camera from the detected lines based on the center localization code includes:

[0086] Line detection is performed on the image of the target region to determine at least two subsets of edge pixels; least squares line fitting is performed on each subset of edge pixels to determine at least two candidate lines; two reference lines are determined based on the distance between the at least two candidate lines and the center positioning code; one of the two reference lines is determined as the target reference line.

[0087] Specifically, for each camera, based on the target region observed by the camera, the corresponding grayscale image region is determined in the grayscale image acquired by the camera, and line detection is performed on the grayscale image region corresponding to the target region to determine at least two edge pixel subsets corresponding to at least two preliminarily detected lines.

[0088] In some embodiments, the process of performing line detection on an image of a target region to determine at least two subsets of edge pixels may include:

[0089] Edge detection is performed on the grayscale image region corresponding to the target region to determine the set of edge pixels. This set includes edge pixels belonging to different straight lines within the grayscale image region corresponding to the target region. A Hough transform is then used to detect straight lines within the edge pixel set, determining at least two sets of line feature parameters corresponding to at least two initial straight lines. Specifically, the Hough transform detects which straight lines may exist within the edge pixel set, as well as the direction and position of each line, thereby determining at least two initial straight lines and a set of line feature parameters corresponding to each initial straight line (each set of line feature parameters includes the direction and position of the corresponding line). Based on each set of line feature parameters, a subset of edge pixels belonging to each initial straight line can be determined within the edge pixel set, thus identifying at least two subsets of edge pixels.

[0090] For example, the distance between each edge pixel in the edge pixel set and the initial line corresponding to the set of line feature parameters can be determined according to each set of line feature parameters, and a subset of edge pixels belonging to the initial line can be selected from the edge pixel set according to the distance.

[0091] For example, the grayscale image region corresponding to the target region can be as follows: Figure 4 The area within the dashed box is shown. The Hough transform detection algorithm can be used to detect lines in this region, yielding at least two initial lines. The Hough transform maps edge points in the image space to the Hough parameter space. And by using a voting mechanism, peak values ​​in the parameter space are found, thereby detecting straight lines. The angle between the normal to the line and the positive X-axis of the camera's image coordinate system is used to describe the direction of the line. It represents the vertical distance from the origin of the camera's image coordinate system to the line, and is used to describe the position of the line.

[0092] In some embodiments, the Hough transform method is robust to noise and outliers, but its accuracy may be limited by the parameter space resolution. To further improve the accuracy of line detection, embodiments of this application perform least-squares line fitting on at least two initial lines detected by the Hough transform to fine-tune and correct the at least two initial lines, thereby determining at least two candidate lines.

[0093] In some embodiments, with the goal of minimizing the sum of squared distances from edge pixels to candidate lines, line fitting is performed based on each subset of edge pixels to determine candidate lines corresponding to each subset of edge pixels, thereby determining at least two candidate lines.

[0094] The optimization objective of least squares line fitting is: for a set of points on a given initial line... Find a straight line ,by The normalization condition is to minimize the sum of the squared distances from all points to this line. This optimization objective can be expressed by formula (1), as follows:

[0095] Formula (1)

[0096] In formula (1), The values ​​range from 1 to n. The constraint condition is Through formula (1), in Under the constraints, the solution makes The minimum values ​​of A, B, and C can then be determined based on the values ​​of A, B, and C and the equation of the line. Determine the candidate lines obtained by least-squares line fitting from the initial line.

[0097] For example, principal component analysis can be used to solve formula (1) by calculating the covariance matrix of the point set and solving for the eigenvector corresponding to its smallest eigenvalue, thus obtaining the values ​​of parameters A, B, and C of the linear equation, as follows:

[0098] 1. Calculate the data center of the point set P: 2. Construct the covariance matrix as shown in formula (2), as follows:

[0099] Formula (2)

[0100] 3. Calculate the eigenvector (A, B) corresponding to the smallest eigenvalue of the covariance matrix according to formula (2). This eigenvector represents the direction of the unit normal vector of the candidate line obtained after least squares line fitting.

[0101] 4. Based on the property that the optimal straight line must pass through the data center, calculate the value of the constant term C in the equation of the straight line. .

[0102] Thus, based on the solved values ​​of A, B, and C, a candidate straight line is determined. For each of the at least two initial lines, the candidate lines corresponding to each initial line can be obtained by solving the problem using formula (1) based on the point set P on each initial line. Thus, at least two candidate lines can be determined by least squares line fitting.

[0103] Furthermore, based on the distances between at least two candidate lines and the center positioning code, two mutually perpendicular reference lines are determined from the at least two candidate lines. In some embodiments, the two perpendicularly intersecting candidate lines closest to the center positioning code can be determined from the at least two candidate lines as the two reference lines. The center positioning code provides the location of the origin of the reference coordinate system, and the two mutually perpendicular reference lines closest to the center positioning code provide two orthogonal direction vectors. Therefore, by selecting the two perpendicularly intersecting candidate lines closest to the center positioning code as reference lines, an accurate, stable, and direct coordinate system axial reference is provided for determining the camera's reference coordinate system. For example, as shown... Figure 5 As shown, among at least two candidate lines detected in the target area, the two perpendicularly intersecting lines 20 and 22 closest to the upper left corner of the center positioning code can be selected as two reference lines.

[0104] In this embodiment, to establish a globally unified coordinate system across all camera calibration results, a unique reference line needs to be determined from two reference lines as a reference coordinate axis (e.g., the X-axis or Y-axis) for the world coordinate system. In some embodiments, a target reference line can be selected from the two reference lines to determine the reference coordinate axis based on the known positional relationship between the preset physical orientation of the positioning code on the target and the two reference lines. For example, each positioning code has a preset corner point (e.g., the lower right corner) that identifies its orientation. Based on the orientational relationship of this corner point relative to the intersection of the two lines, a selection rule can be predefined, such as selecting the line located on a specific side of this corner point (e.g., above the intersection of the lower right corner) as the target reference line. This ensures that different cameras assign axial directions to the same physical line, thus obtaining a consistent reference coordinate system. In other words, at least one camera selects the same reference line as the target reference line, meaning that at least one camera has the same target reference line.

[0105] In this embodiment, the target reference line is used to determine the coordinate axes of the reference coordinate system. For example, at least one camera may select line 22 as the target reference line, which is used to determine the X-axis of the reference coordinate system. At least one camera may also select line 21 as the target reference line to determine the Y-axis of the reference coordinate system. The specific selection depends on the actual situation, and this embodiment does not limit the choice.

[0106] It is understood that the embodiments of this application use Hough transform to stably detect the position of a straight line, and use least squares to perform high-precision estimation of the line parameters within a local range, thereby obtaining a straight line equation with sub-pixel accuracy. This provides a more accurate reference direction for subsequently establishing a high-precision target coordinate system. Thus, highly robust and high-precision straight line detection is achieved, thereby improving the accuracy of further camera calibration based on the straight line detection results.

[0107] In some embodiments, the extrinsic data of each camera includes: rotation and translation matrices from the reference coordinate system defined by the target to the camera coordinate system of each camera; the determination of the extrinsic data of each camera based on the intrinsic data of each camera, the reference plane, and the target reference line in S105 above includes:

[0108] Based on the intrinsic parameter data of each camera, points on the target reference line are mapped to the reference plane to determine the set of three-dimensional coordinate points corresponding to the target reference line. The set of three-dimensional coordinate points includes the three-dimensional coordinate points corresponding to the origin of the reference coordinate system. The origin of the reference coordinate system is determined based on the intersection of the two reference lines. Based on the normal vector of the three-dimensional coordinate point set and the reference plane of each camera, the rotation matrix of each camera is determined. Based on the rotation matrix and the three-dimensional coordinate points corresponding to the origin of the reference coordinate system, the translation matrix of each camera is determined.

[0109] In this embodiment, the intersection of two reference lines is calculated and used as the origin of the reference coordinate system. It can be understood that the reference lines include the target reference line, meaning the origin of the reference coordinate system lies on the target reference line. The target reference line is detected from a two-dimensional grayscale image, and the coordinates of a point on the target reference line are its two-dimensional coordinates in the camera's image coordinate system. The reference plane is obtained by fitting three-dimensional point cloud data. Based on the camera's intrinsic parameters, the two-dimensional coordinates of the points on the target reference line can be mapped to the three-dimensional reference plane, calculating the set of three-dimensional coordinate points corresponding to the target reference line. This set of three-dimensional coordinate points also includes the three-dimensional coordinate points corresponding to the origin obtained by mapping the origin of the reference coordinate system to the origin of the reference plane.

[0110] In some embodiments, the three-dimensional coordinates of the intersection points of the light rays containing points on the target reference line and the reference plane can be calculated based on camera intrinsic data, thereby obtaining the set of three-dimensional coordinate points corresponding to the target reference line. For example, the camera's intrinsic data, such as the intrinsic parameter matrix, can be represented as follows: ;in,( , () represents the origin coordinates of the image coordinate system, that is, the pixel position in the image plane corresponding to the center of the camera's optical axis; , The x and y axes represent the scaling factors of the image along the X and Y axes, respectively, which are the scaling factors that convert distances in the physical world into image pixel coordinates. It can be seen that the intrinsic parameter matrix K defines the projection geometry from the 3D camera coordinate system to the 2D image pixel coordinate system.

[0111] Based on the intrinsic parameter matrix K, we can first transform a two-dimensional coordinate point on the target reference line to the camera coordinate system, and then map the coordinates of that point in the camera coordinate system to the three-dimensional reference plane to determine the corresponding three-dimensional coordinate point. In this way, we can determine the set of three-dimensional coordinate points corresponding to the target reference line.

[0112] For example, for a two-dimensional coordinate point on the target reference line Based on the intrinsic parameter matrix K, it can be transformed to the camera coordinate system using formulas (3) and (4) to obtain the point. The corresponding coordinates in the camera coordinate system as follows:

[0113] Formula (3)

[0114] Formula (4)

[0115] Among them, point The equation of the straight line of the ray can be expressed by formula (5), as follows:

[0116] Formula (5)

[0117] Combining formula (5) with the plane equation of the reference plane The point can be calculated using formulas (6) and (7). The intersection point of the ray and the reference plane is as follows:

[0118] Formula (6)

[0119] Formula (7)

[0120] In formula (6), a, b, c, and d are known parameters in the plane equation of the reference plane. In formula (7), For point The point where the ray intersects the reference plane.

[0121] Thus, by using the above method, we can obtain the three-dimensional coordinate points of each two-dimensional coordinate point on the target reference line mapped to the three-dimensional reference plane, thereby determining the set of three-dimensional coordinate points corresponding to the target reference line.

[0122] In some embodiments, the process of determining the rotation matrix of each camera based on the normal direction of the three-dimensional coordinate point set and the reference plane includes:

[0123] Line fitting and direction extraction are performed based on a set of 3D coordinate points to determine the first coordinate axis of the reference coordinate system; the second coordinate axis of the reference coordinate system is determined based on the normal vector of the reference plane; the third coordinate axis of the reference coordinate system is determined based on the first and second coordinate axes; and the rotation matrix of each camera is determined based on the first, second, and third coordinate axes.

[0124] In this embodiment, a three-dimensional coordinate point set corresponding to the target reference line is used to perform line fitting and line direction extraction in three-dimensional space. The fitted line is used as the first coordinate axis of the reference coordinate system, and the extracted line direction is used as the positive direction of the first coordinate axis. For example, the first coordinate axis of the reference coordinate system can be the X-axis, which can be represented by a three-dimensional normalized vector x_axis.

[0125] In this embodiment, the normal direction of the reference plane is used as the positive direction of the second coordinate axis of the reference coordinate system. For example, the second coordinate axis of the reference coordinate system can be the Z-axis, and the reference coordinate system can use the unit normal vector of the reference plane. To express.

[0126] In this embodiment, the third coordinate axis of the reference coordinate system can be determined by cross-productting the vector corresponding to the first coordinate axis and the vector corresponding to the second coordinate axis. For example, the third coordinate axis may include the Y-axis, which is obtained by cross-productting the x-axis and z-axis. .

[0127] In this embodiment of the application, given that the first coordinate axis, the second coordinate axis, and the third coordinate axis are determined, a rotation matrix from the reference coordinate system to the camera coordinate system can be constructed based on the first coordinate axis, the second coordinate axis, and the third coordinate axis.

[0128] For example, the column vectors of the three coordinate axes are concatenated into The matrix is ​​obtained. According to formula (8), the rotation matrix of each camera from the camera coordinate system to the reference coordinate system is obtained as follows:

[0129] Formula (8)

[0130] In formula (8), Rotation matrix for each camera.

[0131] In this embodiment of the application, given the rotation matrix of each camera, the translation matrix corresponding to each camera can be obtained by combining the rotation matrix with the three-dimensional coordinate points corresponding to the origin of the reference coordinate system mentioned above, using formula (9), as follows:

[0132] Formula (9)

[0133] In formula (9), The translation matrix corresponding to each camera.

[0134] Through the above process, the calibration of each camera in at least one camera is completed, and the rotation and translation matrices of each camera are determined as its extrinsic parameter data. Thus, for a point in the camera coordinate system... The coordinates of the point in the reference coordinate system can be calculated using the rotation and translation matrices of the corresponding camera, combined with formula (10). :

[0135] Formula (10)

[0136] It is understood that, in this embodiment, by placing the target within the overlapping field of view of multiple cameras, only one data acquisition is needed to uniformly map all cameras to the reference coordinate system on the target, thus completing the calibration of multiple cameras. For cases where the overlapping field of view of the cameras is small or the target is partially occluded, as long as the camera can see any one of the four white areas in the target and the ArUco code at the center, the fitted reference plane and two intersecting reference lines on the reference plane obtained in the above calibration process can be obtained, thereby completing the calibration. This solves the problems of low calibration efficiency, large actual accuracy error (especially in the depth direction), susceptibility to noise interference, and insufficient target occlusion or overlapping field of view of cameras in traditional calibration methods, improving the accuracy of camera extrinsic parameter data calibration. When performing coordinate transformation using the camera extrinsic parameter data determined by the calibration method of this embodiment, a smaller actual geometric error can be achieved, and the calibration results are suitable for measurement applications with higher accuracy requirements.

[0137] In some embodiments, after determining the extrinsic parameter data for each camera, the accuracy of the obtained extrinsic parameter data can also be verified by the following methods:

[0138] At least one camera in this application embodiment includes at least a first camera and a second camera. Using the extrinsic parameter data of the first camera, a first spatial point in the point cloud data acquired by the first camera can be transformed from the first camera coordinate system to a reference coordinate system to determine a first mapped spatial point. Using the extrinsic parameter data of the second camera, the first mapped spatial point can be transformed from the reference coordinate system to the second camera coordinate system to determine a second mapped spatial point. The camera calibration accuracy result is determined based on the distance between the second mapped spatial point and the second spatial point.

[0139] In this embodiment, the first camera and the second camera collect data on the same target in the same calibration scene. Therefore, by establishing the correspondence between the three-dimensional coordinate points in the point cloud data collected by the first camera and the three-dimensional coordinate points in the point cloud data collected by the second camera, the spatial points collected by the first camera and the second camera in their respective camera coordinate systems can be found. Thus, based on the spatial points corresponding to the same physical spatial point, a pair of verification points is obtained to verify the accuracy of the calibration, and the extrinsic parameter data obtained from the calibration is verified.

[0140] In this embodiment, the first spatial point and the second spatial point are three-dimensional coordinate points with a pre-established correspondence; the first spatial point and the second spatial point correspond to the same physical spatial point.

[0141] In some embodiments, by identifying pixels in the grayscale images of the first camera and the second camera that correspond to the same physical space point (such as the same corner point of the same positioning code) as matching point pairs, and using these matching point pairs and the intrinsic and extrinsic parameter data of the two cameras, triangulation is performed to obtain the three-dimensional coordinates of the physical space point in a reference coordinate system in three-dimensional space. Transforming these three-dimensional coordinates to the coordinate systems of the first and second cameras respectively yields the corresponding first and second spatial points.

[0142] In some embodiments, point cloud registration methods such as feature matching and iterative nearest point algorithm can be used to register the point cloud data acquired by the first camera and the point cloud data acquired by the second camera, so as to find three-dimensional matching point pairs representing the same physical space point in the point cloud data of the first camera and the point cloud data of the second camera, thereby determining the first space point and the second space point.

[0143] In this embodiment, the camera calibration accuracy is determined based on the distance between two second-mapped spatial points. The distance between two second-mapped spatial points is inversely proportional to the camera calibration accuracy. For example, if the distance between two second-mapped spatial points is greater than or equal to a preset distance threshold, it indicates that the camera calibration accuracy is substandard; if the distance between two second-mapped spatial points is less than the preset distance threshold, it indicates that the camera calibration accuracy is satisfactory.

[0144] It should be noted that, in this embodiment of the application, multiple pairs of three-dimensional verification point pairs can be pre-established. Each pair of three-dimensional verification point pairs includes a first spatial point and a second spatial point corresponding to the same physical spatial point. Based on each pair of three-dimensional verification point pairs, the distance between the second mapped spatial point and the second spatial point is calculated using the above method, thereby obtaining multiple distances. Statistical calculations are performed based on these multiple distances, such as calculating the average and variance, and the camera calibration accuracy is evaluated based on the statistical calculation results.

[0145] Understandably, traditional calibration methods typically use reprojection error to measure calibration quality. However, reprojection error is measured by calculating the error in mapping 3D points to points on a 2D image, and its unit is pixels. This cannot measure actual spatial errors, especially depth deviations, and may result in situations where the overlap error is small while the actual error is large. This application's embodiment uses a geometric error method to verify the calibration results. It calculates the distance between the point where the first spatial point is mapped from the first camera coordinate system to the second camera coordinate system and the corresponding calibration point obtained from the second camera to verify the calibration effect. Geometric error reflects the overall accuracy of the relative pose calibration between the two cameras. When this error, i.e., the distance, exceeds a threshold, it indicates that at least one camera's extrinsic parameter estimation has a deviation.

[0146] In some embodiments, to improve the global consistency of the entire camera system, the geometric errors calculated from multiple sets of verification point pairs observed by all cameras can be used as the overall optimization target. Through bundle adjustment or nonlinear least squares optimization, the extrinsic parameters of all cameras can be jointly optimized, thereby distributing and eliminating errors globally and obtaining consistent and optimal extrinsic parameter estimates.

[0147] This application provides a camera calibration method applicable to real-world scenarios, such as... Figure 6 As shown, it includes:

[0148] S601. Obtain the point cloud image and grayscale image of the target.

[0149] The process involves placing the target at a suitable distance from the camera so that multiple calibrated cameras can see the complete target, thereby acquiring point cloud images (equivalent to point cloud data) and grayscale images.

[0150] S602. Identify the positioning code on the grayscale image to locate the target area observed by the camera.

[0151] For each camera, the white area (equivalent to the target area) and the position of the black bar (containing a straight line) observed by the camera are located by recognizing the ArUco code on the grayscale image.

[0152] S603. Perform plane fitting on the point cloud data of the target area to obtain the plane equation of the reference plane.

[0153] Specifically, point cloud data of the white area in the point cloud image is obtained, and the corresponding points are extracted for RANSAC plane fitting.

[0154] Plane fitting can be performed using the RANSAC algorithm, with the following steps:

[0155] 1. Randomly select three non-collinear points from the point cloud.

[0156] 2. Calculate the plane parameters based on three non-collinear points.

[0157] 3. Calculate the distance from all points to the plane, and designate points that are less than the threshold as interior points.

[0158] 4. Repeat the above steps, update the number of iterations, set the distance of points with a distance greater than the threshold to the threshold value, select the plane with the smallest average distance between all points and the plane, and thus obtain the plane equation of the reference plane.

[0159] S604. Perform line detection and fitting on the grayscale image to obtain the linear equation of the target reference line.

[0160] Specifically, a straight line detection fitting is performed on the target region located by ArUco code on the acquired grayscale image.

[0161] To balance robustness and high accuracy, a combination of Hough transform detection and least squares algorithm is adopted: Hough transform detection is responsible for stably finding the position of the line, while the least squares algorithm provides high-precision estimation of the line parameters within a local range. The detection steps are as follows:

[0162] I. Image Preprocessing.

[0163] For example, grayscale images can be filtered and regions of interest (ROIs) extracted.

[0164] 2. Perform edge detection on the grayscale image region corresponding to the target region to determine the set of edge pixels. For example, the Canny algorithm can be used for edge detection.

[0165] 3. Perform Hough transform detection based on the edge pixel set to obtain the line parameters ( , ).

[0166] in,( , This is equivalent to a set of linear feature parameters corresponding to the initial straight line mentioned above. In this embodiment, at least two initial straight lines and a set of linear feature parameters corresponding to each initial straight line can be obtained through Hough transform detection.

[0167] 4. For each initial straight line, based on the Hough transform detection results, select a subset of edge point pixels near the initial straight line.

[0168] Fifth, perform least-squares line fitting on the subset of edge pixels, which is equivalent to refining the coarse detection results of the Hough transform using the least-squares algorithm.

[0169] 6. Output the refined line parameters of each initial line to determine at least two candidate lines.

[0170] Using the above-mentioned line detection and fitting method, for at least two candidate lines found in the target region, find the two intersecting lines closest to the upper left corner of the ArUco code in the middle, such as line 20 and line 22, so as to obtain two reference lines and their corresponding line equations.

[0171] S605. Calculate the camera's extrinsic parameters using the calculated plane equation and line equation.

[0172] In this step, for lines 20 and 22 found on the grayscale image in the previous step, the intersection point q of the two lines is calculated as the origin of the reference coordinate system, and one of the lines (which line to choose can be determined according to the direction of the ArUco code, and the same line must be chosen for all cameras, such as choosing line 22) is selected as the X-axis of the reference coordinate system.

[0173] Using the camera intrinsic parameters and the coordinates of the points on the detected straight line 22 (two-dimensional image coordinates on the grayscale image), the intersection points of the rays containing these points and the target plane fitted above (three-dimensional coordinates on the point cloud) are calculated, thereby obtaining the three-dimensional coordinates Q corresponding to the origin q obtained from the grayscale image.

[0174] The above method is used to obtain the set of three-dimensional coordinate points mapped from line L1 onto the target plane. Then, a new three-dimensional line fitting is performed using this set of coordinates. The direction of the fitted line is the x-axis direction of the reference coordinate system, denoted as [x-axis direction]. ( It is a three-dimensional normalized vector). The z-axis direction of the reference plane is chosen as the normal direction of the target plane, i.e. The y-axis direction of the reference plane is obtained by the cross product of the x-axis and z-axis, i.e. Construct the rotation matrix from the reference coordinate system to the camera coordinate system; the translation vector can be obtained from the three-dimensional coordinate Q of the origin of the reference coordinate system in the camera coordinate system, thus finally solving for the rotation and translation matrix from the camera coordinate system to the reference coordinate system, obtaining the camera's extrinsic parameter data, and completing the camera calibration.

[0175] It is understood that the embodiments of this application have high calibration efficiency, and the extrinsic parameter calibration of multiple cameras can be completed by collecting data only once. Due to the use of RANSAC plane fitting and line fitting methods, the calibration results are less affected by noise and outliers. The depth information of the calibration points is used, and there are no problems of scale instability or amplification of depth direction error. The calibration results are more accurate and stable. Furthermore, the calibration area can be located according to the ArUco code distributed on the target. Calibration can still be performed when the target is partially occluded or the overlapping area between cameras is small, which improves the robustness of the calibration method.

[0176] This application provides a camera calibration system, including: a target, at least one camera, and a processing unit, wherein the target includes: multiple sub-regions divided by at least two mutually perpendicular straight lines, at least one region positioning code for each sub-region, and a center positioning code for the center of the target.

[0177] Each of at least one camera is used to acquire point cloud and image data of the target, and to determine the grayscale image and point cloud data of each camera;

[0178] The processing unit is used to perform location code recognition on the grayscale image, determine the center location code and at least one target region location code, and determine the target region observed by each camera based on the at least one target region location code; the target region includes at least one target sub-region among multiple sub-regions; perform plane fitting based on the point cloud data corresponding to the target region to determine the reference plane corresponding to each camera; perform line detection on the grayscale image region corresponding to the target region, and determine the target reference line corresponding to each camera from the detected lines based on the center location code; and determine the extrinsic data of each camera based on the intrinsic parameter data of each camera, the reference plane and the target reference line.

[0179] It should be noted that in the above camera calibration system, the processing unit can execute the camera calibration method as described in any of the above embodiments, which will not be repeated here.

[0180] To implement the method of the embodiments of this application, based on the same inventive concept, the embodiments of this application also provide a camera calibration device, such as... Figure 7 As shown, the camera calibration device 1 includes:

[0181] The data acquisition module 11 is used to acquire point cloud and image data of the target through each of at least one camera, and to determine the grayscale image and point cloud data of each camera; the target includes: multiple sub-regions divided by at least two mutually perpendicular straight lines, at least one region positioning code of each sub-region, and a center positioning code of the target center.

[0182] The calibration module 12 is used to perform positioning code recognition on the grayscale image, determine the center positioning code and at least one target region positioning code, and determine the target region observed by each camera based on the at least one target region positioning code; the target region includes at least one target sub-region among multiple sub-regions; perform plane fitting based on the point cloud data corresponding to the target region to determine the reference plane corresponding to each camera; perform line detection on the grayscale image region corresponding to the target region, and determine the target reference line corresponding to each camera from the detected lines based on the center positioning code; and determine the extrinsic data of each camera based on the intrinsic parameter data of each camera, the reference plane and the target reference line.

[0183] In some embodiments, the target is placed in the overlapping field of view of multiple cameras such that each camera can observe at least a center positioning code and at least one area positioning code; the intersection of at least two mutually perpendicular straight lines falls on the center positioning code; at least one area positioning code for each sub-region is distributed circumferentially on the target in each sub-region; each area positioning code corresponds to a unique positioning identifier.

[0184] In some embodiments, the calibration module 12 is further configured to select at least three points from the point cloud data corresponding to the target region each time to perform multiple plane fittings to determine multiple candidate reference planes; and to determine a reference plane among the multiple candidate reference planes based on the average distance from the points in the target region to each candidate reference plane.

[0185] In some embodiments, the calibration module 12 is further configured to determine multiple distance values ​​from multiple points in the target region to each candidate reference plane; update the distance values ​​among the multiple distance values ​​that are greater than a preset distance threshold to the preset distance threshold; and determine the average distance based on the updated multiple distance values.

[0186] In some embodiments, the calibration module 12 is further configured to perform line detection on the image of the target region, determine at least two subsets of edge pixels; perform least squares line fitting based on each subset of edge pixels in the at least two subsets of edge pixels, determine at least two candidate lines; determine two reference lines based on the distance between the at least two candidate lines and the center positioning code; and determine one of the two reference lines as the target reference line.

[0187] In some embodiments, the calibration module 12 is further configured to perform line fitting based on each subset of edge pixels with the objective of minimizing the sum of squared distances from edge pixels to candidate lines, thereby determining candidate lines corresponding to each subset of edge pixels, and thus determining at least two candidate lines.

[0188] In some embodiments, the calibration module 12 is further configured to determine, from at least two candidate lines, two perpendicularly intersecting candidate lines that are closest to the center positioning code, as two reference lines.

[0189] In some embodiments, the extrinsic data of each camera includes: a rotation matrix and a translation matrix from the reference coordinate system defined by the target to the camera coordinate system of each camera; the calibration module 12 is further configured to map points on the target reference line to a reference plane based on the intrinsic data of each camera, and determine the set of three-dimensional coordinate points corresponding to the target reference line; the set of three-dimensional coordinate points includes the three-dimensional coordinate points corresponding to the origin of the reference coordinate system; the origin is determined based on the intersection of two reference lines; the rotation matrix of each camera is determined based on the normal vector of the three-dimensional coordinate point set and the reference plane; and the translation matrix of each camera is determined based on the rotation matrix and the three-dimensional coordinate points corresponding to the origin.

[0190] In some embodiments, the calibration module 12 is further configured to perform line fitting and direction extraction based on a set of three-dimensional coordinate points to determine the first coordinate axis of the reference coordinate system; determine the second coordinate axis of the reference coordinate system based on the normal vector of the reference plane; determine the third coordinate axis of the reference coordinate system based on the first and second coordinate axes; and determine the rotation matrix of each camera based on the first, second, and third coordinate axes.

[0191] In some embodiments, at least one camera includes at least a first camera and a second camera. The camera calibration device further includes a verification module. The verification module is used to determine the extrinsic parameter data of each camera, and then, using the extrinsic parameter data of the first camera, transform the first spatial point in the point cloud data acquired by the first camera from the first camera coordinate system to the reference coordinate system to determine the first mapped spatial point; using the extrinsic parameter data of the second camera, transform the first mapped spatial point from the reference coordinate system to the second camera coordinate system to determine the second mapped spatial point; and determine the camera calibration accuracy result based on the distance between the second mapped spatial point and the second spatial point; the second spatial point and the first spatial point correspond to the same physical spatial point.

[0192] It should be noted that the description of the above device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0193] Based on the hardware implementation of each unit in the aforementioned camera calibration device, this application also provides an electronic device, such as... Figure 8 As shown, the electronic device 90 includes: a processor 901, a memory 902 configured to store computer programs capable of running on the processor, and a bus system 903;

[0194] The processor 901 is configured to execute the method steps in the foregoing embodiments when running a computer program.

[0195] Of course, in practical applications, such as Figure 8 As shown, the various components in the electronic device 90 are coupled together via a bus system 903. It is understood that the bus system 903 is used to enable communication between these components. In addition to a data bus, the bus system 903 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 903 in the figure.

[0196] In practical applications, the aforementioned processor can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field-Programmable Gate Array (FPGA), controller, microcontroller, and microprocessor. It is understood that, for different devices, the electronic device used to implement the above processor function can also be other types, and the embodiments of this application do not specifically limit it.

[0197] The aforementioned memory can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provides instructions and data to the processor.

[0198] Optionally, the electronic device 90 can be a chip, which may further include an input interface. The processor can control the input interface to communicate with other devices or chips; specifically, it can acquire information or data sent by other devices or chips.

[0199] Optionally, the chip may also include an output interface. The processor can control this output interface to communicate with other devices or chips; specifically, it can output information or data to other devices or chips.

[0200] In an exemplary embodiment, this application also provides a computer-readable storage medium, such as a memory including a computer program, which can be executed by a processor of an electronic device to perform the steps of the aforementioned method.

[0201] This application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the methods in this application.

[0202] Optionally, the computer program product can be applied to the electronic device in the embodiments of this application, and the computer program instructions cause the computer to execute the corresponding processes implemented by the electronic device in the various methods of the embodiments of this application. For the sake of brevity, they will not be described in detail here.

[0203] This application also provides a computer program. Optionally, the computer program can be applied to the electronic device in the embodiments of this application. When the computer program is run on a computer, it causes the computer to execute the corresponding processes implemented by the electronic device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0204] It should be understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0205] It should be understood that the terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. The expressions “having,” “may have,” “comprising,” and “including,” or “may include” and “may contain” used herein may be used to indicate the presence of a corresponding feature (e.g., an element such as a numerical value, function, operation, or component), but do not exclude the presence of additional features.

[0206] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and are not necessarily used to describe a specific order or sequence. For example, without departing from the scope of this invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information.

[0207] The technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0208] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatus, and devices can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical, or other forms.

[0209] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0210] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0211] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A camera calibration method, characterized by, include: Point cloud and image acquisition of the target are performed by each of at least one camera, and grayscale images and point cloud data of each camera are determined. The target includes: multiple sub-regions divided by at least two mutually perpendicular straight lines, at least one region positioning code for each sub-region, and a center positioning code for the center of the target. The grayscale image is subjected to location code recognition to determine the center location code and at least one target region location code, and the target region observed by each camera is determined based on the at least one target region location code; the target region includes at least one target sub-region among the plurality of sub-regions; Based on the point cloud data corresponding to the target region, a plane fitting is performed to determine the reference plane corresponding to each camera; Line detection is performed on the grayscale image region corresponding to the target region, and the target reference line corresponding to each camera is determined from the detected lines based on the center positioning code; Based on the intrinsic parameter data of each camera, the reference plane, and the target reference line, the extrinsic parameter data of each camera are determined.

2. The method of claim 1, wherein, in, The target is placed in the overlapping field of view of multiple cameras, such that each camera can observe at least the central positioning code and at least one regional positioning code; the intersection of the at least two mutually perpendicular straight lines falls on the central positioning code; at least one regional positioning code for each sub-region is distributed circumferentially on the target in each sub-region; each regional positioning code corresponds to a unique positioning identifier.

3. The method according to claim 1 or 2, characterized in that, The step of performing plane fitting based on the point cloud data corresponding to the target region to determine the reference plane corresponding to each camera includes: At least three points are selected from the point cloud data corresponding to the target region each time for multiple plane fittings to determine multiple candidate reference planes; The reference plane is determined from the plurality of candidate reference planes based on the average distance from a point in the target region to each candidate reference plane.

4. The method according to claim 3, characterized in that, The method further includes: Determine multiple distance values ​​from multiple points in the target region to each candidate reference plane; The distance values ​​that are greater than a preset distance threshold among the plurality of distance values ​​are updated to the preset distance threshold, and the average distance is determined based on the updated plurality of distance values.

5. The method according to claim 1 or 2, characterized in that, The step of performing line detection on the grayscale image region corresponding to the target region and determining the target reference line corresponding to each camera from the detected lines based on the center localization code includes: Line detection is performed on the image of the target region to determine at least two subsets of edge pixels; Based on each of the at least two edge pixel subsets, least squares line fitting is performed to determine at least two candidate lines; Two reference lines are determined based on the distances between the at least two candidate lines and the center positioning code; One of the two reference lines is determined as the target reference line.

6. The method according to claim 5, characterized in that, The step of performing least-squares line fitting based on each of the at least two subsets of edge pixels to determine at least two candidate lines includes: With the goal of minimizing the sum of squared distances from edge pixels to candidate lines, line fitting is performed based on each subset of edge pixels to determine the candidate lines corresponding to each subset of edge pixels, thereby determining at least two candidate lines.

7. The method according to claim 5, characterized in that, The step of determining two reference lines based on the distances between the at least two candidate lines and the center positioning code includes: From the at least two candidate lines, determine the two perpendicularly intersecting candidate lines that are closest to the center positioning code, and use them as the two reference lines.

8. The method according to claim 5, characterized in that, The extrinsic data for each camera includes: rotation and translation matrices from the reference coordinate system defined by the target to the camera coordinate system of each camera; determining the extrinsic data for each camera based on the intrinsic data of each camera, the reference plane, and the target reference line includes: Based on the intrinsic parameter data of each camera, points on the target reference line are mapped to a reference plane to determine the set of three-dimensional coordinate points corresponding to the target reference line; the set of three-dimensional coordinate points includes the three-dimensional coordinate points corresponding to the origin of the reference coordinate system; the origin is determined based on the intersection of the two reference lines. Based on the normal vectors of the three-dimensional coordinate point set and the reference plane, the rotation matrix of each camera is determined; The translation matrix of each camera is determined based on the rotation matrix and the three-dimensional coordinate points corresponding to the origin.

9. The method according to claim 8, characterized in that, Determining the rotation matrix of each camera based on the normal vector of the three-dimensional coordinate point set and the reference plane includes: Based on the set of three-dimensional coordinate points, straight line fitting and direction extraction are performed to determine the first coordinate axis of the reference coordinate system; The second coordinate axis of the reference coordinate system is determined based on the normal vector of the reference plane; The third coordinate axis of the reference coordinate system is determined based on the first coordinate axis and the second coordinate axis; The rotation matrix of each camera is determined based on the first coordinate axis, the second coordinate axis, and the third coordinate axis.

10. The method according to any one of claims 1, 2, 4, 6-9, characterized in that, The at least one camera includes at least a first camera and a second camera, and after determining the extrinsic data of each camera, the method further includes: Using the extrinsic data of the first camera, the first spatial point in the point cloud data collected by the first camera is transformed from the first camera coordinate system to the reference coordinate system to determine the first mapped spatial point; Using the extrinsic data of the second camera, the first mapped spatial point is transformed from the reference coordinate system to the second camera coordinate system to determine the second mapped spatial point; The camera calibration accuracy is determined based on the distance between the second mapped spatial point and the second spatial point; the second spatial point corresponds to the same physical spatial point as the first spatial point.

11. A camera calibration system, characterized in that, Includes a target, at least one camera, and a processing unit, wherein, The target includes: multiple sub-regions divided by at least two mutually perpendicular straight lines, at least one region positioning code for each sub-region, and a center positioning code for the center of the target. Each of the at least one camera is used to acquire point cloud and image data of the target, and to determine the grayscale image and point cloud data of each camera. The processing unit is configured to perform location code recognition on the grayscale image, determine the center location code and at least one target region location code, and determine the target region observed by each camera based on the at least one target region location code; the target region includes at least one target sub-region among the plurality of sub-regions; perform plane fitting based on the point cloud data corresponding to the target region to determine the reference plane corresponding to each camera; perform line detection on the grayscale image region corresponding to the target region, and determine the target reference line corresponding to each camera from the detected lines based on the center location code; and determine the extrinsic data of each camera based on the intrinsic parameter data of each camera, the reference plane and the target reference line.

12. A camera calibration device, characterized in that, The device includes: The data acquisition module is used to acquire point cloud and image data of the target through each of at least one camera, and to determine the grayscale image and point cloud data of each camera; the target includes: multiple sub-regions divided by at least two mutually perpendicular straight lines, at least one region positioning code for each sub-region, and a center positioning code for the center of the target; A calibration module is used to perform location code recognition on the grayscale image, determine the center location code and at least one target region location code, and determine the target region observed by each camera based on the at least one target region location code; the target region includes at least one target sub-region among the plurality of sub-regions; perform plane fitting based on the point cloud data corresponding to the target region to determine the reference plane corresponding to each camera; perform line detection on the grayscale image region corresponding to the target region, and determine the target reference line corresponding to each camera from the detected lines based on the center location code; and determine the extrinsic data of each camera based on the intrinsic parameter data of each camera, the reference plane and the target reference line.

13. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method of any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, It stores computer-executable instructions or a computer program for causing a processor to execute, thereby implementing the method as described in any one of claims 1 to 10.

15. A computer program product, characterized in that, It includes computer-executable instructions or computer programs that, when executed by a processor, implement the method as described in any one of claims 1 to 10.

Images