Array camera and laser radar external parameter calibration method based on plane mark and medium

Abstract

The invention provides an array camera and laser radar external parameter calibration method based on a plane mark and a medium. The problems of low automation degree, low calibration precision and the like in array camera and laser radar external parameter calibration in related technologies can be solved. The method comprises the following steps: acquiring a sub-camera image containing a plane calibration object and laser radar point cloud data which are synchronously triggered and acquired by an array camera and a laser radar; in the sub-camera image, acquiring a first three-dimensional coordinate set of the hole center of each circular hole in a corresponding sub-camera coordinate system; in the laser radar point cloud data, acquiring a second three-dimensional coordinate set of the center of each circular hole in a laser radar coordinate system; and based on the corresponding relationship between the centers of the circular holes in the first three-dimensional coordinate set and the second three-dimensional coordinate set, obtaining a spatial transformation matrix from the sub-camera coordinate system to the laser radar coordinate system so as to complete external parameter calibration of the corresponding sub-camera and the laser radar.

Description

Technical Field

[0001] This application relates to the field of image data processing, and in particular to a method and medium for extrinsic parameter calibration of array cameras and lidar based on planar markers. Background Technology

[0002] In recent years, intelligentization has become a significant development trend across various fields. Due to inherent physical limitations, single sensors often struggle to comprehensively and accurately describe complex environmental information. The combined use of LiDAR and array cameras effectively compensates for the shortcomings of single sensors, enhancing the system's perception capabilities through data complementarity. This approach has been widely applied in remote sensing, robotics, autonomous driving, and 3D reconstruction. The extrinsic parameter calibration between the array camera and LiDAR aims to accurately acquire the spatial transformation relationship (including rotation matrices and translation vectors) between their coordinate systems, as well as the relative pose relationships between the array camera's sub-cameras. This calibration process is a crucial prerequisite for achieving accurate spatiotemporal alignment and effective fusion of multimodal data.

[0003] Furthermore, array cameras, through horizontal or vertical camera arrangement, combine high imaging accuracy with wide field of view coverage. Accurate extrinsic parameter calibration allows for the estimation of the overall physical resolution of the stitched and fused large field-of-view image, as well as the field-of-view overlap rate between sub-cameras at each viewpoint, thus providing clear imaging parameters and accuracy data support for practical engineering applications.

[0004] Existing technical solutions can be mainly divided into two categories: one is based on specific calibration objects (such as checkerboards, 3D calibration boards, etc.), which calculate extrinsic parameters by extracting corresponding feature points from images and point clouds respectively. Since array cameras acquire two-dimensional image texture information, while lidar acquires three-dimensional point cloud spatial structures, the data modal differences between the two are significant, making it difficult to directly establish a correspondence. Usually, it is necessary to rely on manually labeling pairs of feature points with the same name. The other category is targetless natural scene calibration methods, which rely on common features such as edges and planes in the environment for calibration. However, the calibration accuracy of this type of method is usually low in complex scenes. Summary of the Invention

[0005] This application provides a method and medium for extrinsic parameter calibration of array cameras and lidar based on planar markers, which can solve the problems of low automation and low calibration accuracy in the extrinsic parameter calibration of array cameras and lidar in related technologies.

[0006] To achieve the above objectives, this application adopts the following technical solution: Firstly, a method for extrinsic parameter calibration of an array camera and lidar based on planar markers is provided, including: Acquire sub-camera images and LiDAR point cloud data containing a planar calibration object, which are synchronously triggered and collected by the array camera and LiDAR; wherein, the surface of the planar calibration object is provided with at least one ArUco code and multiple circular holes, the circular holes on the planar calibration object are arranged asymmetrically, the geometric relationship between the corner point of the ArUco code and the center of the circular hole is known, and the planar calibration object is placed in the common field of view area of ​​the sub-camera of the array camera to be calibrated and the LiDAR; In the sub-camera image, based on the known geometric relationship between the ArUco code corner points and the center of the circular hole, the first three-dimensional coordinate set of each circular hole center in the corresponding sub-camera coordinate system is obtained; Based on the ArUco code, a planar calibration object point cloud is obtained from the lidar point cloud data. The point cloud of the planar calibration object is fitted with circular holes to obtain the second three-dimensional coordinate set of the center of each circular hole in the lidar coordinate system; Based on the correspondence between the centers of the circular holes in the first three-dimensional coordinate set and the second three-dimensional coordinate set, the spatial transformation matrix from the sub-camera coordinate system to the lidar coordinate system is obtained to complete the external parameter calibration of the corresponding sub-camera and the lidar.

[0007] Secondly, a device for calibrating the extrinsic parameters of an array camera and lidar based on planar markers is provided, comprising: The first acquisition module is used to acquire sub-camera images and lidar point cloud data containing a planar calibration object, which are synchronously triggered and collected by the array camera and lidar. The planar calibration object has at least one ArUco code and multiple circular holes on its surface. The circular holes on the planar calibration object are arranged asymmetrically. The geometric relationship between the corner points of the ArUco code and the center of the circular holes is known. The planar calibration object is placed within the common field of view area of ​​the sub-camera of the array camera to be calibrated and the lidar. The second acquisition module is used to acquire, based on the known geometric relationship between the ArUco code corner point and the center of the circular hole in the sub-camera image, the first three-dimensional coordinate set of each circular hole center in the corresponding sub-camera coordinate system; The third acquisition module is used to obtain the plane calibration object point cloud from the lidar point cloud data based on the ArUco code; The fourth acquisition module is used to perform circular hole fitting on the point cloud of the planar calibration object to obtain the second three-dimensional coordinate set of the center of each circular hole in the lidar coordinate system. The fifth acquisition module is used to acquire the spatial transformation matrix from the sub-camera coordinate system to the lidar coordinate system based on the correspondence between the centers of the circular holes in the first three-dimensional coordinate set and the second three-dimensional coordinate set, so as to complete the external parameter calibration of the corresponding sub-camera and the lidar.

[0008] Thirdly, an electronic device is provided, characterized in that it comprises: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect.

[0009] Fourthly, a computer program product is provided, comprising a computer program or instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect.

[0010] The beneficial effects of this invention are: This invention provides a planar calibration method for the extrinsic parameters of an array camera and a lidar system. The method first designs a dedicated planar calibration object suitable for the joint calibration of the array camera and lidar; then extracts the coordinates of feature points from the array camera image and the lidar point cloud, respectively; finally, by mapping the lidar point cloud to a two-dimensional image based on intensity information, image features are identified, and the extrinsic parameter calibration matrix between the array camera and the lidar is solved. This calibration method solves the problems of low automation and low calibration accuracy in the extrinsic parameter calibration of array cameras and lidar systems in related technologies. Attached Figure Description

[0011] Figure 1 The diagram illustrates the steps of an array camera and lidar extrinsic parameter calibration method based on planar markers.

[0012] Figure 2 The illustration shows a design of a marker with a flat sign.

[0013] Figure 3 The illustration shows the ArUco code with ID 0 in the DICT_4x4 dictionary.

[0014] Figure 4 A schematic diagram of a semi-circular array camera and lidar is shown.

[0015] Figure 5 Schematic illustration Figure 4 Diagram showing the conversion relationship between the central lidar and sub-camera 1.

[0016] Figure 6 The diagram illustrates the steps of a method for stitching array camera parameters.

[0017] Figure 7 The diagram schematically illustrates a cylindrical spatial image created by an array camera and lidar.

[0018] Figure 8This is a schematic diagram of image grid division.

[0019] Figure 9 The illustration shows the rough cylindrical imaging effect of the array camera.

[0020] Figure 10 The illustration shows the cylindrical imaging effect of the array camera.

[0021] Figure 11 The schematic diagram shows the cylindrical projection unfolded view of the array camera.

[0022] Figure 12 An example diagram illustrating the spatial geometry parameters of an array camera is shown.

[0023] Figure 13 A block diagram of an array camera and lidar extrinsic parameter calibration device based on planar markers is shown schematically.

[0024] Figure 14 A block diagram of a stitching device for array camera parameters is shown schematically.

[0025] Figure 15 A block diagram of an electronic device is shown schematically.

[0026] Figure 16 A block diagram of a computer-readable medium is shown schematically. Detailed Implementation

[0027] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0028] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0029] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0030] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0031] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any and one or more of the listed applications.

[0032] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of exemplary embodiments, and the modules or processes in the drawings are not necessarily essential for implementing this application, and therefore cannot be used to limit the scope of protection of this application.

[0033] This invention provides a planar calibration method for the extrinsic parameters of an array camera and a lidar based on planar markers. The method first designs a dedicated planar calibration object suitable for the joint calibration of the array camera and lidar; extracts the coordinates of feature points from the array camera image and the lidar point cloud respectively; identifies image features by mapping the lidar point cloud to a two-dimensional image according to intensity information, and establishes pairs of corresponding feature points; finally, it uses the pairs of corresponding feature points to solve for the extrinsic parameter calibration matrix between the array camera and the lidar. According to the first specific embodiment of this invention, as... Figure 1 As shown, this invention provides a method for extrinsic parameter calibration of an array camera and lidar based on planar markers, including: Step S11: Acquire the sub-camera image containing the planar calibration object and the lidar point cloud data, which are synchronously triggered and collected by the array camera and lidar.

[0034] The planar calibration object has at least one ArUco code and multiple circular holes on its surface. The circular holes on the planar calibration object are arranged asymmetrically. The geometric relationship between the corner points of the ArUco code and the center of the circular holes is known. The planar calibration object is placed within the common field of view area of ​​the sub-camera of the array camera to be calibrated and the lidar.

[0035] To obtain a high-precision extrinsic parameter calibration matrix for lidar and array cameras, this invention designs a planar marker for array cameras and lidar calibration. For example... Figure 2 As shown, in one specific embodiment, the calibrator is 1400mm long and 1000mm wide, and includes two types of planar marking features: one is an ArUco code of type DICT_4X4, and the other is a planar circular hole; wherein the side length of the ArUco code and the diameter of the circular hole are both 200mm.

[0036] The center position uses an asymmetrical layout to facilitate rapid determination of the correspondence; the ArUco code uses the DICT_4x4 dictionary for better detection results.

[0037] This invention is in Figure 2 and Figure 3 In the specific embodiment shown, a DICT_4x4 dictionary is used, and its marker consists of an inner coding area and an outer border. The inner coding area is a 4*4 binary grid containing 16 black and white squares, used to encode unique ID information. The outer border, a black border (one square wide) surrounding the coding area, mainly assists the detector in achieving rapid marker localization and orientation recognition. Figure 3 The example shown is the standard ArUco tag with ID 0 in the DICT_4x4 dictionary.

[0038] To facilitate understanding of the technical solution of this invention, the data acquisition and calibration principles of lidar and array camera are briefly explained with reference to the accompanying drawings. Lidar measures the distance between the target and the lidar receiver by emitting a beam of light, acquiring accurate three-dimensional point cloud data. It is widely used in obstacle detection, real-time map construction, autonomous navigation, and other fields. The point cloud data acquired by lidar is characterized by high accuracy and strong anti-interference capabilities, is unaffected by changes in lighting, and contains accurate depth information and rich intensity information.

[0039] An array camera is an imaging system consisting of multiple sub-cameras (imaging units) arranged in a specific space. These sub-cameras have the same or similar optical parameters and work together to form an overall imaging system. The core feature of an array camera is that, while maintaining the high-resolution imaging capability of each sub-camera, the spatial distribution of multiple sub-cameras significantly expands the overall field of view of the system. Figure 4 One specific embodiment shown employs an array camera system consisting of seven sub-cameras, arranged along a semi-circular arc space. For example... Figure 4 As shown in the schematic diagram of the layout of the semi-circular array camera and the lidar working together, cameras numbered 1 to 7 are the sub-camera units that make up this array.

[0040] The principle of relative pose calibration between lidar and array camera sub-cameras is as follows: Figure 5 As shown (in this invention, sub-camera 1 is used as an example in subsequent examples), the calibration system composed of lidar and sub-cameras includes four coordinate systems, namely the lidar coordinate system. Sub-camera coordinate system Pixel coordinate system and world coordinate system The transformation parameters between the coordinate points in the lidar coordinate system and the coordinate points in the sub-camera coordinate system are the extrinsic parameters of both. The calculation of the extrinsic parameters relies on their shared features (i.e., features with the same name). Common features include point, line, and surface features, as well as mutual information.

[0041] Let (u,v) be the image pixel coordinates, (xc,yc,zc) be the point in the sub-camera coordinate system, and (xl,yl,zl) be the point in the lidar coordinate system. The sub-camera intrinsic parameter is K, which is known data. , The rotation and translation matrices of the lidar and sub-camera are unknown. The principle of lidar and sub-camera extrinsic parameter calibration is as follows: (1) The transformation relationship between the sub-camera coordinate system and the image coordinate system is as follows: (2) The transformation relationship between the lidar coordinate system and the sub-camera coordinate system is as follows: (3) When calibrating the extrinsic parameters of an array camera system, each sub-camera and its corresponding lidar must be calibrated in pairs to obtain the independent transformation matrix of each sub-camera relative to the lidar coordinate system. These separately calibrated transformation matrices together constitute the complete set of extrinsic parameters between the entire array camera system and the lidar.

[0042] This invention establishes a precise spatial correspondence between each sub-camera in the array camera and the lidar by separately calibrating the coordinate transformation relationship (i.e., extrinsic parameter matrix) between them. The following explanation uses sub-camera 1 as an example to illustrate the extrinsic parameter calibration method between the lidar and the camera.

[0043] The core of extrinsic parameter calibration lies in extracting the coordinates of the same set of physical features in the lidar coordinate system and the sub-camera coordinate system from the synchronously acquired point cloud data and sub-camera image data. This invention selects the circular holes on the calibration plate as stable features. By detecting the center points of the circular holes in the point cloud and the images respectively, a set of corresponding feature points is obtained, and then the extrinsic parameter matrix between the lidar and sub-camera 1 is solved based on these corresponding points.

[0044] After acquiring the sub-camera images and LiDAR point cloud data, proceed to step S12.

[0045] Step S12: In the sub-camera image, based on the known geometric relationship between the ArUco code corner points and the center of the circular holes, obtain the first three-dimensional coordinate set of each circular hole center in the corresponding sub-camera coordinate system.

[0046] In the image data, the pixel coordinates and ID of each ArUco code corner point can be detected quickly and accurately, and the pixel coordinates of the corner points can be transformed into the coordinate system of sub-camera 1. Based on the geometric and physical relationship between the ArUco code and the circular hole, the coordinates of the center of the circular hole in the coordinate system of sub-camera 1 can be obtained.

[0047] Specifically, step S12 may include the following steps: Step S121: In the sub-camera image, detect the ArUco code to obtain the coordinates of the ArUco code corner point in the image pixel coordinate system.

[0048] Step S121 may include the following steps: Step S1211: Perform grayscale conversion and filtering preprocessing on the sub-camera image; Step S1212: Perform edge detection and contour extraction on the processed sub-camera image to filter out quadrilateral contours; Step S1213: Perform perspective correction on the quadrilateral outline to obtain a square image; Step S1214: The corrected square image is gridded, and the grid brightness value is obtained to generate binary code; Step S1215: Match the binary code with a predefined ArUco dictionary to obtain the ID of the ArUco code; Step S1216: Output the pixel coordinates and IDs of the four corner points of the ArUco code.

[0049] Step S122: Based on the intrinsic parameters of the sub-camera, obtain the three-dimensional coordinates of the ArUco code corner point in the corresponding sub-camera coordinate system.

[0050] Step S123: Based on the known geometric relationship between the corner coordinates of the ArUco code and the center of the circular hole, obtain the first three-dimensional coordinate set of each circular hole center in the corresponding sub-camera coordinate system.

[0051] ArUco codes are detected using digital image processing methods. Each ArUco code is identified from the calibration object image, and its corresponding four corner pixel coordinates and ID are output. Figure 4 and Figure 5 In one specific embodiment, the process of obtaining the first three-dimensional coordinate set includes the following ArUco code detection process, corner coordinate calculation in the coordinate system of sub-camera 1, and hole center coordinate calculation: (1) The ArUco code detection process includes: 1) Image preprocessing: Convert the image to grayscale and apply Gaussian blur filter to reduce noise interference; 2) Contour extraction: Image binarization is performed using adaptive threshold Canny edge detection to extract all closed contours, and invalid contours are filtered out based on area size to reduce computational load; 3) Polygon approximation and filtering: Perform polygon approximation on each contour and filter out candidate contours that have 4 vertices and are convex polygons; 4) Perspective correction: For each candidate quadrilateral, calculate the perspective transformation matrix and correct it to a standard square image; 5) ID recognition: Divide the corrected square image into an N×N grid (this invention uses a DICT_4X4 dictionary, N=4), read the brightness value of the internal grid and binarize it (0 / 1) to generate a binary code; match it with the predefined DICT_4X4 dictionary and output the corresponding ID. If the match is successful, the ArUco code recognition is deemed valid. 6) Corner point sorting: The four detected corner points are arranged in a fixed order. The order adopted in this invention is: upper left, upper right, lower right, and lower left of the calibration object design pattern.

[0052] (2) The calculation process of corner coordinates in the sub-camera 1 coordinate system includes: based on formula (2), combined with the intrinsic parameters of sub-camera 1, the pixel coordinates of each ArUco code corner point are transformed from the image coordinate system to the sub-camera 1 coordinate system, and finally the set of ArUco code corner coordinates in the sub-camera 1 coordinate system is obtained.

[0053] (3) Calculation of the coordinates of the circular hole centers: Based on the known geometric relationship between the circular hole centers and the ArUco code corner points, the three-dimensional coordinates of each circular hole center in the coordinate system of sub-camera 1 can be calculated. The calculation process includes: 1) Establishment of the theoretical coordinate system of the calibration object: Based on the geometric layout of the calibration object design drawing, take the upper left corner of the calibration object as the origin, the length direction as the x-axis and the width direction as the y-axis, establish the theoretical coordinate system of the calibration object, and obtain the coordinates of the center of each ordered circular hole and the corner point of the ArUco code in this system.

[0054] 2) Coordinate system mapping: Based on the ArUco code ID, obtain the corresponding positions of the corner points in the coordinate system of the sub-camera 1 in the theoretical coordinate system of the calibration object, and form matching point pairs; use these corresponding point pairs to calculate the coordinate transformation matrix from the theoretical coordinate system of the calibration object to the coordinate system of the sub-camera 1.

[0055] 3) Calculation of three-dimensional coordinates of the center of the circular hole: Apply the coordinate transformation matrix of the previous step to transform the coordinates of the center of each circular hole to the coordinate system of sub-camera 1, and obtain an ordered set of coordinates of the center of the circular holes in this system.

[0056] After obtaining the first set of three-dimensional coordinates, proceed to step S13.

[0057] Step S13: Based on the ArUco code, obtain the plane calibration object point cloud from the lidar point cloud data.

[0058] Specifically, step S13 may include the following steps: Step S131: In the lidar point cloud data, the point cloud is partitioned according to the scanning angle and mapped into a two-dimensional image.

[0059] Specifically, step S131 may include the following steps: Step S1311: With the origin of the lidar as the center, the scanning space is divided into multiple fan-shaped areas at preset angle intervals; Step S1312: For the point cloud in each sector area, based on the azimuth and pitch coordinates of the point cloud and combined with the preset scaling factor, the point cloud is projected onto a two-dimensional image with the same size as the camera image, and the point cloud intensity value is assigned as the gray value of the corresponding pixel.

[0060] Step S132: Detect ArUco code in the two-dimensional image.

[0061] Step S133: After detecting the ArUco code in the two-dimensional image, the spatial range of the planar calibration object in the lidar point cloud is calculated based on the known position of the ArUco code in the planar calibration object.

[0062] Specifically, step S133 may include the following steps: Step S1331: Restore the detected ArUco code corner points to the lidar coordinate system according to the scaling factor of the sector region where they are located; Step S1332: Based on the known geometric relationship between the ArUco code corner point and the center of the circular hole, obtain an ordered set of approximate coordinates of the center of the circular hole and the ArUco code corner point in the lidar coordinate system. Step S1333: Determine the spatial range of the planar calibration object in the lidar point cloud based on the maximum and minimum coordinate values ​​in the approximate coordinate set.

[0063] Step S134: Based on the calculation results, crop the point cloud within the spatial range to obtain the calibration point cloud.

[0064] Step S14: Perform circular hole fitting on the point cloud of the planar calibration object to obtain the second three-dimensional coordinate set of the center of each circular hole in the lidar coordinate system.

[0065] Specifically, step S14 may include the following steps: Step S141: Extract edge points based on curvature from the point cloud of the calibration object plane; Step S142: Cluster the edge points; Step S143: Perform three-dimensional circle fitting on each point cloud cluster using the least squares method to obtain the coordinate set of the center of the circular hole in the lidar coordinate system.

[0066] Given that the field of view of lidar is usually wide, ranging from tens to hundreds of meters, while the size of the calibration object is relatively small, it is very difficult to obtain the coordinates of the center of the circular hole on the calibration object directly from the original point cloud generated by lidar. Therefore, it is necessary to crop the lidar point cloud to the calibration object point cloud and then use the point cloud processing algorithm to calculate the coordinates of the center of the circular hole in the lidar coordinate system.

[0067] Combination Figure 4 and Figure 5 In one specific embodiment, the process of obtaining the second three-dimensional coordinate set includes the following calibration object point cloud extraction process and point cloud circular hole center coordinate extraction process: (1) Point cloud extraction process for calibration objects: 1) During the lidar calibration data acquisition process, since the calibration object is always facing the lidar, this invention divides the point cloud into 90° zones based on the lidar scanning angle range (if the total scanning angle is less than 90°, no zoning is needed). A two-dimensional image plane is constructed for each zone of the point cloud, and the point cloud intensity values ​​are mapped to the image plane. The ArUco code in the image is detected using data processing methods to determine the approximate range of the calibration object in the point cloud. The specific steps are as follows: 2) Point cloud angle partitioning: Centered on the origin of the lidar, the point cloud is divided into multiple partitions at 90° intervals and 45° intervals (for example, the 360° scanning range can be divided into the following 8 partitions: {(0~90°],(45~135°],(90~180°],(135~225°],(180~270°],(225~315°],(270~360°],(315~360+45°]}); 3) Point cloud to image mapping: Create a 2D image with the same size as the image of sub-camera 1 for each partition point cloud, and map the center of the point cloud to the center pixel of the image; calculate the scaling factor according to the size of the partition point cloud, and map the intensity value of each point to the 2D image point by point according to the distance to the origin from far to near; 4) ArUco code detection: ArUco code detection is performed sequentially on the partitioned two-dimensional images. When an ArUco code is successfully detected in an image, the positions of the remaining codes can be deduced through geometric relationships, without the need to continue detecting the remaining images. 5) Coordinate inverse calculation: The detected ArUco code corner points are restored to the lidar coordinate system according to the scaling factor of the partition to obtain their approximate positions; combined with the coordinate relationship between the center of the circular hole and the ArUco code corner points in the theoretical coordinate system of the calibration object, an ordered set of approximate coordinates of the center of the circular hole and the ArUco code corner points in the lidar coordinate system is obtained. 6) Cropping the calibration object point cloud: Based on the maximum and minimum coordinate values ​​in the coordinate set obtained in the previous step, determine the spatial range of the calibration object in the original point cloud, and crop the point cloud within this range as the calibration object point cloud.

[0068] (2) Process for extracting the center coordinates of circular holes in point cloud: 1) After obtaining the point cloud of the calibration object, the coordinates of the center of the circular hole in the lidar coordinate system can be calculated using point cloud processing algorithms. The specific process is as follows: 2) Planar point cloud extraction: The point cloud is fitted to a plane according to a preset plane threshold, and the plane point cloud of the calibration object is segmented according to the plane equation parameters. Outliers are removed to obtain clean planar point cloud data. 3) Edge feature extraction: Calculate the curvature of each point and set a curvature threshold to identify edge feature points; 4) Point cloud clustering: Based on the preset clustering radius and minimum number of cluster points, adjacent points are grouped into the same cluster by judging whether the Euclidean distance between points is less than the clustering radius; 5) Hole center coordinate fitting: For each point cloud cluster, the least squares method is used to perform 3D circle fitting. Combined with the known diameter of the circular hole, the circular holes that meet the conditions are selected to obtain the set of hole center coordinates in the lidar coordinate system. Furthermore, the hole center coordinates can be sorted: the above hole center coordinate set is matched with the approximate coordinate set of the hole center and ArUco code corner point of the circular hole in the lidar coordinate system to obtain an ordered sequence of hole center coordinates in the lidar coordinate system.

[0069] After obtaining the second set of three-dimensional coordinates, proceed to step S15.

[0070] Step S15: Based on the correspondence between the centers of the circular holes in the first three-dimensional coordinate set and the second three-dimensional coordinate set, obtain the spatial transformation matrix from the sub-camera coordinate system to the lidar coordinate system to complete the extrinsic parameter calibration of the corresponding sub-camera and the lidar.

[0071] In one embodiment, step S15 may include the following steps: Step S1511: Perform distance matching between the coordinate set of the center of the circular hole in the lidar coordinate system and the approximate coordinate set to obtain an ordered sequence of coordinates of the center of the circular hole in the lidar coordinate system. Step S1512: Perform coordinate sorting and matching between the first three-dimensional coordinate set and the second three-dimensional coordinate set to establish correct pairs of corresponding points; Step S1513: Solve the spatial transformation matrix from the sub-camera coordinate system to the lidar coordinate system using the singular value decomposition algorithm to complete the extrinsic parameter calibration of the corresponding sub-camera and lidar.

[0072] Based on the coordinate set of the circular aperture center in the sub-camera 1 coordinate system and the ordered coordinate set of the circular aperture center in the lidar coordinate system, a pair of corresponding feature points between the sub-camera 1 and the lidar coordinate system is generated. Then, the transformation matrix from the lidar to the sub-camera 1 coordinate system is solved by SVD (singular value decomposition). This matrix is ​​the calibration extrinsic parameter between the lidar and the sub-camera 1.

[0073] In another embodiment, step S15 may include the following steps.

[0074] Step S1521: Obtain the coordinate sequence of the center of all circular holes of the object under the coordinate system of the lidar and the coordinate system of camera 1 respectively.

[0075] Step S1522: Solve the spatial transformation matrix from the sub-camera coordinate system to the lidar coordinate system using the iterative nearest point algorithm to complete the extrinsic parameter calibration of the corresponding sub-camera and the lidar.

[0076] Because the circular hole layout of the calibration object designed in this invention is asymmetrical, when extracting the center coordinates of all the circular holes in the design drawing, it is not necessary to sort the positions of the hole centers. The ICP (Iterative Closest Point) algorithm can be directly used to calculate the transformation matrix between the lidar and the sub-camera 1 coordinate system. This matrix is ​​the calibration extrinsic parameter between the lidar and the sub-camera 1. The steps of the ICP algorithm are as follows: 1) Initialization: Set the initial transformation matrix to the identity matrix, and input the source point cloud and the target point cloud; 2) Iteration: Each iteration performs the following steps: a. Nearest neighbor matching: For each point in the source point cloud, find the nearest neighbor in the target point cloud and establish a set of corresponding point pairs; b. Transformation estimation: Based on the set of corresponding point pairs established in the previous step, calculate the optimal rigid body transformation matrix that minimizes the mean square error between corresponding point pairs; c. Apply transformation: Update the source point cloud based on the optimal rigid body transformation matrix; d. Convergence detection: If the change in the transformation matrix or the change in the error function is less than a preset threshold, or the maximum number of iterations is reached, then the iteration is stopped; 3) Output result: Output the spatial transformation matrix from the sub-camera coordinate system to the lidar coordinate system obtained in step 2).

[0077] This invention first designs a dedicated planar calibration object suitable for the joint calibration of array cameras and lidar; extracts the coordinates of feature points from the array camera image and the lidar point cloud respectively; and identifies image features and solves the extrinsic parameter calibration matrix between the array camera and lidar by mapping the lidar point cloud to a two-dimensional image according to intensity information. This calibration method solves the problems of low automation and low calibration accuracy in the extrinsic parameter calibration of array cameras and lidar in related technologies.

[0078] Furthermore, after completing the extrinsic parameter calibration of the corresponding sub-cameras and the lidar, the extrinsic parameter calibration matrix can be used for array camera parameter stitching and array camera spatial parameter calculation. Through accurate extrinsic parameter calibration, the overall physical resolution of the large field-of-view image obtained after stitching and fusion, as well as the field-of-view overlap rate between sub-cameras at each viewpoint, can be estimated using the calibration parameters, thereby providing clear imaging parameters and accuracy data support for practical engineering applications. According to the second specific embodiment of the present invention, please refer to... Figure 6 , Figure 6 The diagram schematically illustrates the steps of a method for stitching array camera parameters. For example... Figure 6 As shown, the present invention also provides a method for stitching array camera parameters, the stitching method being based on Figure 1 The extrinsic parameter matrix between each sub-camera and the lidar obtained through calibration, i.e., the spatial transformation matrix from the coordinate system of each sub-camera to the coordinate system of the lidar, is stitched together using the following method: Step S21: For the image acquired by each sub-camera in the array camera, the image is divided into multiple image blocks, and the pixel coordinates of the corner points of each image block are extracted.

[0079] exist Figure 1 After completing the extrinsic parameter calibration of each sub-camera and lidar in the array camera and obtaining the independent spatial transformation matrix of each camera, the parameters can be stitched together so that any pixel in each image data acquired by the array camera can be transformed into the lidar coordinate system by back-projecting the camera model onto the camera coordinate system to form a directional ray that starts from the optical center of the camera, passes through the pixel and points to the three-dimensional space.

[0080] This invention applies an array camera system, already calibrated with lidar extrinsic parameters, to the task of photographing spatial structures. Taking a cylindrical space as an example (its inner surface is referred to as the cylinder surface), the imaging scene is as follows: Figure 7 As shown, the outer area represents the point cloud data of the cylindrical space collected by the lidar, while the middle part is a schematic diagram of the shooting angles and field of view distribution of each sub-camera in the array camera. During the array camera parameter stitching process, the process of projecting the image pixels of each sub-camera onto the cylinder can be decomposed into two main geometric operations: finding the intersection of the camera pixel rays with the cylinder and selecting the intersection points.

[0081] Step S22: Using the extrinsic parameter matrix corresponding to the sub-camera to which each image block corner point belongs, project the image block corner point onto the lidar coordinate system, and calculate the coordinates of the intersection point of each image block corner point with the target cylinder in the lidar coordinate system.

[0082] The geometric parameters of the target cylindrical surface are obtained from point cloud data collected by lidar on the same scene.

[0083] Specifically, step S22 may include the following steps: Step S221: Project each image block corner point onto a three-dimensional ray in the coordinate system of its corresponding sub-camera; Step S222: Using the extrinsic parameter matrix of the sub-camera, the three-dimensional ray is transformed to the lidar coordinate system; Step S223: Solve the mathematical equations of the transformed three-dimensional ray and the target cylinder to obtain the intersection parameters; Step S224: If there are two intersection points, select the intersection point located in front of the camera as the valid projection point.

[0084] The cylindrical geometric parameters (cylinder axis direction) can be obtained from the lidar point cloud data. A point c on the axis of the cylinder, and the radius R of the cylinder.

[0085] For each sub-camera, its pixel is first back-projected onto a 3D ray in the camera coordinate system through the camera model (starting from the camera optical center and passing through the 3D direction corresponding to the pixel). This ray is then transformed into the lidar coordinate system through the array camera and lidar extrinsic parameter matrix set calculated in Section 4, and its intersection with a specified cylinder is calculated.

[0086] Let the transformed ray parameter equation be: (4) in, The position of the camera optical center in the lidar coordinate system. This represents the normalized pixel ray direction in the lidar coordinate system.

[0087] The square of the distance from the point to the axis of the cylinder is: (5) Let it equal to We obtain a quadratic equation in terms of t: (6) The coefficients are: (7) Solve the discriminant ,like The ray has no intersection with the cylinder; if Then we get two solutions. : (8) Corresponding to two intersection points : (9) Since the ray may intersect the cylinder at two points (one near and one far), the visible intersection point (located in front of the camera and usually the near point) needs to be selected. The selection method is as follows: The two intersection points in the lidar coordinate system Based on the transformation back to the camera coordinate system using the extrinsic parameter matrix, the two intersection points in the camera coordinate system are obtained. Check the intersection point coordinates and depth coordinates. If one point is satisfied >0 (in front of the camera), another point If <0 (behind the camera), then take Points with a depth coordinate greater than 0 are considered valid projection points; if the depth coordinates of two points have the same sign, the pixel may be invalid (e.g., when a ray tangentially passes over a cylindrical surface).

[0088] After obtaining a valid projection point, proceed to step S23.

[0089] Step S23: Based on the coordinates of the cylindrical intersection points of the image patch corners of all sub-cameras, perform image drawing in the lidar coordinate system to obtain the initial stitched image.

[0090] The images from each sub-camera of the array camera are divided into an M*N grid, with each grid representing an image block, such as... Figure 8 As shown. For each image patch acquired by the array camera's sub-cameras, the pixel coordinates of its four corner points are first extracted. Then, according to step S22, the cylindrical projection points of these corner points in the lidar coordinate system are calculated. By plotting the projection points of all corner points on the cylinder, a rough cylindrical imaging effect of the array camera can be obtained, as shown. Figure 9 As shown.

[0091] Step S24: The overlapping areas in the initial stitched image are processed by an image fusion algorithm to generate a seamless stitched image of the target cylinder.

[0092] Optionally, the image fusion algorithm is a Poisson fusion algorithm.

[0093] To further generate a complete and seamless image, this invention treats each image block as a single unit. After projecting its four corner points onto the cylindrical surface, these projection points collectively form a closed quadrilateral region. By sequentially filling the quadrilateral regions corresponding to all image blocks, a continuous and seamless complete image can be formed on the cylindrical surface, as shown in the image. Figure 10 As shown.

[0094] like Figure 8 As shown, the shooting angles of the sub-cameras in the array camera differ, resulting in varying image content and perspectives. Direct projection can easily lead to noticeable boundary lines in the overlapping areas. To address this issue, this invention employs a Poisson fusion algorithm to fuse overlapping regions, effectively eliminating boundary lines and creating a natural, seamless transition in the overall image. The final complete image output by the array camera after cylindrical projection and unfolding is shown in the figure. Figure 11 .

[0095] The Poisson fusion algorithm is implemented as follows: 1) Preprocessing of target and source images: Locating the target image region to be fused and its boundaries Extract the corresponding content from the source image; 2) Constructing the Poisson equation: In Internally establish equations Where v is the gradient field of the source image, and div represents the divergence of the gradient field. Let f be the Laplace operator, and f be the fusion result to be determined. 3) Boundary condition setting: The pixel values ​​of the target image are used as Dirichlet boundary conditions; 4) Discretization solution: The Poisson equation is discretized into a system of linear equations using methods such as finite difference, and then solved using iterative methods (such as Gauss-Seidel) or direct methods; 5) Result synthesis: Substitute the solved f into the target region. It blends seamlessly with the background area.

[0096] Step S25: Calculate the spatial geometric parameters of the array camera based on the seamlessly stitched image.

[0097] An array camera integrates individual sub-cameras, and by calibrating parameters, it can accurately estimate the spatial geometric parameters of the image, providing data support for the engineering applications of array cameras. These spatial geometric parameters include at least the ground sampling distance and spatial coverage of each sub-camera, as well as the overall spatial coverage and spatial overlap of the array camera.

[0098] (1) Calculation of GSD for each sub-camera of the array camera Ground Sample Distance (GSD) refers to the actual distance on the ground corresponding to a single pixel in a remote sensing image, and is a key indicator for measuring the spatial resolution of an image. In this invention, GSD is used to quantify the spatial scale represented by each pixel in the image. The magnitude of GSD directly determines the image's detail resolution capability: the smaller the GSD value, the richer the details the image can present; conversely, the larger the GSD, the more blurred the details. Therefore, GSD is essentially a core parameter that balances imaging coverage and detail accuracy, and has a direct impact on image analysis and applications.

[0099] Set the width of the sub-camera image to W and the height to H, and the sub-camera index. The GSD calculation steps are as follows: Calculate the horizontal length of a single image patch Take the two left corner points of the image block and calculate the Euclidean distance between their cylindrical projection points in the horizontal direction, which is the average length of the image block in the horizontal direction. Calculate the vertical length of a single image patch Take the two corner points on the upper side of the image block and calculate the Euclidean distance between their cylindrical projection points in the vertical direction, which is the average length of the image block in the vertical direction. Calculate the GSD of a single image patch: (10) in, and The width and height of the image block.

[0100] (2) Spatial coverage of each sub-camera of the array camera Calculate the horizontal coverage of the sub-camera Take the left corner of the first image block in the first row of the sub-camera grid image and the right corner of the last image block. Calculate the projection points of the two corner points onto the cylindrical surface in the lidar coordinate system. Calculate the Euclidean distance between the two projection points in the horizontal direction, which is the horizontal coverage range of the sub-camera. Calculate the vertical coverage of the sub-camera Take the top corner of the first image block in the first column of the sub-camera grid image and the bottom corner of the last image block. Calculate the projection points of the two corner points onto the cylindrical surface in the lidar coordinate system. Calculate the Euclidean distance between the two projection points in the vertical direction, which is the vertical coverage range of the sub-camera. (3) Spatial coverage of array camera Calculate the horizontal coverage angle of the sub-camera Take the left corner of the first image block in the first row of the sub-camera grid image and the right corner of the last image block. Calculate the projection points of the two corner points onto the cylinder in the lidar coordinate system. Project them onto the cross section where point c is located on the axis of the cylinder. Calculate the angle between the rays from point c to the two projection points. This is the coverage angle of the sub-camera in the horizontal direction. Calculate the coverage angle of the array camera Take the left corner of the first image block in the first row of the first sub-camera of the array camera and the right corner of the last image block in the last sub-camera. Calculate the projection points of the two corner points onto the cylinder in the lidar coordinate system. Project them onto the cross section where point c is located on the axis of the cylinder. Calculate the angle formed by the rays from point c to the two projection points, which is the coverage angle of the array camera in the horizontal direction. Computational array camera curved space coverage : (11) (4) Spatial overlap rate of array cameras Calculate the overlap ratio Ratio: The number of sub-cameras in the array camera is n. (12) An example of calculating the spatial geometric parameters of the array camera in this invention, such as... Figure 12 As shown. From top to bottom on the left are the GSD of each sub-camera of the array camera, the spatial coverage range of each sub-camera of the array camera, the coverage range of the array camera, the coverage angle of the array camera, and the overlap rate of the array camera; on the right is the projection imaging effect of each sub-camera of the array camera onto the cylindrical surface of the checkerboard pattern.

[0101] This invention proposes a generalizable array camera data parameter stitching method. After solving the extrinsic parameter calibration matrix between the array camera and the lidar, the extrinsic parameter calibration matrix is ​​used for array camera parameter stitching and array camera spatial parameter calculation, including: estimating the imaging accuracy of the stitched image data, calculating the spatial coverage and overlap rate, and providing parameter support for array camera design and selection.

[0102] According to a third specific embodiment of the present invention, such as Figure 13 As shown, the present invention provides an array camera and lidar extrinsic parameter calibration device 100 based on planar markers, comprising: The first acquisition module 110 is used to acquire sub-camera images and lidar point cloud data containing a planar calibration object, which are synchronously triggered and collected by the array camera and lidar. The planar calibration object has at least one ArUco code and multiple circular holes on its surface. The circular holes on the planar calibration object are arranged asymmetrically. The geometric relationship between the corner point of the ArUco code and the center of the circular hole is known. The planar calibration object is placed in the common field of view area of ​​the sub-camera of the array camera to be calibrated and the lidar. The second acquisition module 120 is used to acquire, based on the known geometric relationship between the ArUco code corner point and the center of the circular hole in the sub-camera image, the first three-dimensional coordinate set of each circular hole center in the corresponding sub-camera coordinate system; The third acquisition module 130 is used to acquire the plane calibration object point cloud from the lidar point cloud data based on the ArUco code; The fourth acquisition module 140 is used to perform circular hole fitting on the point cloud of the planar calibration object to obtain the second three-dimensional coordinate set of the center of each circular hole in the lidar coordinate system. The fifth acquisition module 150 is used to acquire the spatial transformation matrix from the sub-camera coordinate system to the lidar coordinate system based on the correspondence between the centers of the circular holes in the first three-dimensional coordinate set and the second three-dimensional coordinate set, so as to complete the external parameter calibration of the corresponding sub-camera and the lidar.

[0103] According to a fourth specific embodiment of the present invention, such as Figure 14 As shown, the present invention provides a stitching device 100' for array camera parameters, comprising: The image processing module 110' is used to divide the image acquired by each sub-camera in the array camera into multiple image blocks and extract the pixel coordinates of the corner points of each image block; The first calculation module 120' is used to project the corner points of each image block onto the lidar coordinate system using the extrinsic parameter matrix corresponding to the sub-camera to which each corner point belongs, and to calculate the coordinates of the intersection point of each corner point of the image block with the target cylinder in the lidar coordinate system; wherein, the geometric parameters of the target cylinder are obtained from the point cloud data collected by the lidar from the same scene; The image drawing module 130' is used to draw images in the lidar coordinate system based on the cylindrical intersection coordinates of the corner points of the image blocks of all sub-cameras, so as to obtain the initial stitched image. The image generation module 140' is used to process the overlapping areas in the initial stitched image through an image fusion algorithm to generate a seamless stitched image of the target cylinder; The second calculation module 150' is used to calculate the spatial geometric parameters of the array camera based on the seamlessly stitched image.

[0104] According to a fifth specific embodiment of the present invention, the present invention provides an electronic device, such as... Figure 15 As shown, Figure 15 This is a block diagram illustrating an electronic device according to an exemplary embodiment.

[0105] The following reference Figure 15 To describe an electronic device 200 according to this embodiment of the present application. Figure 15The electronic device 200 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0106] like Figure 15 As shown, the electronic device 200 is presented in the form of a general-purpose computing device. The components of the electronic device 200 may include, but are not limited to: at least one processing unit 210, at least one storage unit 220, a bus 230 connecting different system components (including storage unit 220 and processing unit 210), a display unit 240, etc.

[0107] The storage unit stores program code that can be executed by the processing unit 210, causing the processing unit 210 to perform the steps described in this specification according to various exemplary embodiments of this application. For example, the processing unit 210 can perform actions such as... Figure 1 or Figure 6 The steps are shown in the figure.

[0108] The storage unit 220 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 2201 and / or a cache storage unit 2202, and may further include a read-only memory unit (ROM) 2203.

[0109] The storage unit 220 may also include a program / utility 2204 having a set (at least one) program module 2205, such program module 2205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0110] Bus 230 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0111] Electronic device 200 can also communicate with one or more external devices 200' (e.g., keyboard, pointing device, Bluetooth device, etc.), enabling users to communicate with devices that interact with electronic device 200, and / or any device that allows electronic device 200 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 250. Furthermore, electronic device 200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 260. Network adapter 260 can communicate with other modules of electronic device 200 via bus 230. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0112] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware.

[0113] Therefore, according to a fourth specific embodiment of the present invention, the present invention provides a computer-readable medium. For example... Figure 16 As shown, the technical solution according to the embodiments of the present invention can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.) or on a network, and includes several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the above-described method according to the embodiments of the present invention.

[0114] The software product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0115] The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0116] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0117] The aforementioned computer-readable medium carries one or more programs, which, when executed by a device, cause the computer-readable medium to perform the functions of the first embodiment.

[0118] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0119] Through the description of the above embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions of the embodiments of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, portable hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of the present invention.

[0120] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for extrinsic parameter calibration of an array camera and lidar based on planar markers, characterized in that, include: Acquire sub-camera images and LiDAR point cloud data containing a planar calibration object, which are synchronously triggered and collected by the array camera and LiDAR; wherein, the surface of the planar calibration object is provided with at least one ArUco code and multiple circular holes, the circular holes on the planar calibration object are arranged asymmetrically, the geometric relationship between the corner point of the ArUco code and the center of the circular hole is known, and the planar calibration object is placed in the common field of view area of ​​the sub-camera of the array camera to be calibrated and the LiDAR; In the sub-camera image, based on the known geometric relationship between the ArUco code corner points and the center of the circular hole, the first three-dimensional coordinate set of each circular hole center in the corresponding sub-camera coordinate system is obtained; Based on the ArUco code, the point cloud of the calibration object is obtained from the lidar point cloud data. The point cloud of the planar calibration object is fitted with circular holes to obtain the second three-dimensional coordinate set of the center of each circular hole in the lidar coordinate system; Based on the correspondence between the centers of the circular holes in the first three-dimensional coordinate set and the second three-dimensional coordinate set, the spatial transformation matrix from the sub-camera coordinate system to the lidar coordinate system is obtained to complete the external parameter calibration of the corresponding sub-camera and the lidar.

2. The method according to claim 1, characterized in that, The step of obtaining the first three-dimensional coordinate set of the center of each circular hole in the corresponding sub-camera coordinate system includes: In the sub-camera image, the ArUco code is detected to obtain the coordinates of the ArUco code corner point in the image pixel coordinate system; Based on the intrinsic parameters of the sub-camera, obtain the three-dimensional coordinates of the ArUco code corner point in the corresponding sub-camera coordinate system; Based on the known geometric relationship between the corner coordinates of the ArUco code and the center of the circular hole, the first three-dimensional coordinate set of each circular hole center in the corresponding sub-camera coordinate system is obtained.

3. The method according to claim 2, characterized in that, The step of detecting the ArUco code to obtain the coordinates of the ArUco code corner points in the image pixel coordinate system includes: The sub-camera images are preprocessed by grayscale conversion and filtering; Edge detection and contour extraction are performed on the processed sub-camera images to filter out quadrilateral contours; Perspective correction is performed on the quadrilateral outline to obtain a square image; The corrected square image is meshed, and the mesh brightness values ​​are obtained to generate binary codes; The binary code is matched with a predefined ArUco dictionary to obtain the ID of the ArUco code; Output the pixel coordinates and IDs of the four corner points of the ArUco code.

4. The method according to claim 3, characterized in that, The acquisition of the calibration point cloud by the plane includes: In the lidar point cloud data, the point cloud is partitioned according to the scanning angle and mapped into a two-dimensional image; Detect ArUco codes in the two-dimensional image; After detecting the ArUco code in the two-dimensional image, the spatial range of the planar calibration object in the lidar point cloud is calculated based on the known position of the ArUco code in the planar calibration object. The point cloud within the spatial range is cropped based on the calculation results to obtain the calibration object point cloud.

5. The method according to claim 4, characterized in that, The step of partitioning the point cloud according to the scanning angle and mapping it to a two-dimensional image includes: Centered on the origin of the lidar, the scanning space is divided into multiple fan-shaped areas at preset angular intervals; For the point cloud in each sector area, based on the azimuth and pitch coordinates of the point cloud and combined with the preset scaling factor, the point cloud is projected onto a two-dimensional image with the same size as the camera image, and the point cloud intensity value is assigned as the gray value of the corresponding pixel.

6. The method according to claim 5, characterized in that, After detecting the ArUco code in the two-dimensional image, the spatial range of the planar calibration object in the lidar point cloud is calculated based on the known position of the ArUco code in the planar calibration object, including: The detected ArUco code corner points are restored to the lidar coordinate system according to the scaling factor of the sector region where they are located; Based on the known geometric relationship between the ArUco code corner point and the center of the circular hole, an ordered set of approximate coordinates of the center of the circular hole and the ArUco code corner point in the lidar coordinate system is obtained. The spatial range of the planar calibration object in the lidar point cloud is determined based on the maximum and minimum coordinate values ​​in the approximate coordinate set.

7. The method according to claim 6, characterized in that, The step of fitting circular holes to the point cloud of the planar calibration object to obtain a second three-dimensional coordinate set of the center of each circular hole in the lidar coordinate system includes: Curvature-based edge point extraction is performed on the point cloud of the calibration object; Cluster the edge points; For each point cloud cluster, a three-dimensional circle is fitted using the least squares method to obtain the coordinate set of the center of the circular hole in the lidar coordinate system.

8. The method according to claim 7, characterized in that, The step of obtaining the spatial transformation matrix from the sub-camera coordinate system to the lidar coordinate system based on the correspondence between the centers of the circular holes in the first and second three-dimensional coordinate sets, in order to complete the extrinsic parameter calibration of the corresponding sub-camera and the lidar, includes: The coordinate set of the center of the circular hole in the lidar coordinate system is matched with the approximate coordinate set to obtain an ordered sequence of coordinates of the center of the circular hole in the lidar coordinate system. The first three-dimensional coordinate set and the second three-dimensional coordinate set are sorted and matched to establish correct pairs of corresponding points; The spatial transformation matrix from the sub-camera coordinate system to the lidar coordinate system is solved by the singular value decomposition algorithm to complete the extrinsic parameter calibration of the corresponding sub-camera and lidar.

9. The method according to claim 7, characterized in that, The step of obtaining the spatial transformation matrix from the sub-camera coordinate system to the lidar coordinate system based on the correspondence between the centers of the circular holes in the first and second three-dimensional coordinate sets, in order to complete the extrinsic parameter calibration of the corresponding sub-camera and the lidar, includes: Obtain the coordinate sequence of the center of all circular holes of the object under the coordinate system of the lidar and the coordinate system of camera 1 respectively; The spatial transformation matrix from the sub-camera coordinate system to the lidar coordinate system is solved by the iterative nearest point algorithm to complete the extrinsic parameter calibration of the corresponding sub-camera and lidar.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program or instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-9.

Images