Multi-sensor joint calibration method, computer equipment and storage medium

Abstract

The embodiment of the invention relates to the technical field of external parameter calibration, and provides a multi-sensor joint calibration method, computer equipment and a storage medium. The method comprises the steps that a calibration reference object image, a first point cloud data set and a second point cloud data set are obtained, the calibration reference object image is obtained by collecting a target scene provided with a calibration reference object through a first sensor, and the first point cloud data set is obtained by scanning the target scene through a three-dimensional scanner; the second point cloud data set is obtained by scanning a target scene by a second sensor, and the resolution ratio of the first point cloud data set is higher than that of the second point cloud data set; determining a first external parameter matrix from the three-dimensional scanner to the first sensor according to the calibration reference object image and the first point cloud data set; determining a second external parameter matrix from the second sensor to the three-dimensional scanner according to the first external parameter matrix, the first point cloud data set and the second point cloud data set; and determining a target external parameter matrix from the second sensor to the first sensor according to the first external parameter matrix and the second external parameter matrix. According to the embodiment of the invention, the accuracy of multi-sensor joint calibration of external parameters is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of external parameter calibration, and particularly to a multi-sensor joint calibration method, a computer device, and a storage medium. Background Art

[0002] In the positioning, mapping, and navigation tasks of autonomous mobile devices, multiple sensors such as cameras and radars are often used to make up for each other's deficiencies, improving the accuracy of positioning and mapping and the accuracy of environmental perception. Usually, before using multiple sensors such as cameras and radars, it is necessary to calibrate the external parameters between multiple sensors offline. For example, calibrate the external parameters from the radar to the camera.

[0003] Currently, the external parameters from the radar to the camera are mainly calibrated based on the calibration board. The external parameter calibration method based on the calibration board needs to extract the features of the calibration board from the image data of the calibration board collected by the camera and extract the features of the calibration board from the point cloud data obtained by the radar scanning the calibration board. Then, based on the extracted features of the calibration board, the external parameters from the radar to the camera are calibrated.

[0004] However, the point cloud data obtained by the radar scanning the calibration board is sparse, and it is difficult to extract the calibration board from the sparse point cloud data, and the accuracy of the extraction result cannot be guaranteed, resulting in inaccurate external parameters from the radar to the camera obtained by calibration. Therefore, how to improve the accuracy of multi-sensor joint calibration of external parameters is an urgent problem to be solved at present. Summary of the Invention

[0005] Embodiments of the present invention provide a multi-sensor joint calibration method, a computer device, and a storage medium, aiming to improve the accuracy of multi-sensor joint calibration of external parameters.

[0006] In a first aspect, an embodiment of the present invention provides a multi-sensor joint calibration method, including:

[0007] Obtain a calibration reference object image, a first point cloud data set, and a second point cloud data set. The calibration reference object image is obtained by a first sensor collecting a target scene provided with the calibration reference object. The first point cloud data set is obtained by a 3D scanner scanning the target scene. The second point cloud data set is obtained by a second sensor scanning the target scene. The resolution of the first point cloud data set is higher than the resolution of the second point cloud data set;

[0008] Determine a first external parameter matrix from the 3D scanner to the first sensor according to the calibration reference object image and the first point cloud data set;

[0009] Determine a second external parameter matrix from the second sensor to the 3D scanner according to the first external parameter matrix, the first point cloud data set, and the second point cloud data set;

[0010] Determine the target extrinsic matrix from the second sensor to the first sensor according to the first extrinsic matrix and the second extrinsic matrix.

[0011] In a second aspect, an embodiment of the present invention further provides a computer device, which includes a processor, a memory, a computer program stored on the memory and executable by the processor, and a data bus for realizing the connection and communication between the processor and the memory. When the computer program is executed by the processor, the multi-sensor joint calibration method described in the first aspect is implemented.

[0012] In a third aspect, an embodiment of the present invention further provides a storage medium for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the multi-sensor joint calibration method described in the first aspect.

[0013] An embodiment of the present invention provides a multi-sensor joint calibration method, a computer device, and a storage medium. Based on a calibration reference object image and a first point cloud dataset, the embodiment of the present invention determines a first extrinsic matrix from a 3D scanner to a first sensor, and then based on the first extrinsic matrix, the first point cloud dataset, and a second point cloud dataset, determines a second extrinsic matrix from a second sensor to the 3D scanner. Finally, according to the first extrinsic matrix and the second extrinsic matrix, the target extrinsic matrix from the second sensor to the first sensor is determined. The entire process does not involve extracting the features of the calibration reference object from a point cloud dataset with low resolution (sparse point cloud dataset), eliminating the influence of feature extraction errors on the calibration result, thereby improving the accuracy of the extrinsic parameters of multi-sensor joint calibration. Description of the Drawings

[0014] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0015] Figure 1 is a schematic flowchart of a multi-sensor joint calibration method provided by an embodiment of the present invention;

[0016] Figure 2 is an example diagram of a calibration reference object image in an embodiment of the present invention;

[0017] Figure 3 is a point cloud map corresponding to the second point cloud dataset in an embodiment of the present invention;

[0018] Figure 4 isFigure 1 Schematic diagram of the sub-step process of the multi-sensor joint calibration method in

[0019] Figure 5 is Figure 4 Schematic diagram of the sub-step process of the multi-sensor joint calibration method in

[0020] Figure 6 Schematic block diagram of a computer device provided by an embodiment of the present invention. Detailed implementation manners

[0021] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0022] The flowcharts shown in the accompanying drawings are only illustrative examples, and do not necessarily include all contents and operations / steps, nor do they necessarily need to be executed in the described order. For example, some operations / steps can also be decomposed, combined, or partially merged. Therefore, the actual execution order may change according to the actual situation.

[0023] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in this specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0024] Currently, the external parameters from the radar to the camera are mainly calibrated based on the external parameter calibration method of the calibration board. The external parameter calibration method based on the calibration board needs to extract the features of the calibration board from the image data of the calibration board collected by the camera and from the point cloud data obtained by the radar scanning the calibration board, and then calibrate the external parameters from the radar to the camera based on the extracted features of the calibration board. However, the point cloud data obtained by the radar scanning the calibration board is sparse, and it is difficult to extract the calibration board from the sparse point cloud data, and the accuracy of the extraction result cannot be guaranteed, resulting in inaccurate external parameters from the radar to the camera calibrated. Therefore, how to improve the accuracy of the multi-sensor joint calibration of external parameters is an urgent problem to be solved at present.

[0025] To solve the above problems, an embodiment of the present invention provides a multi-sensor joint calibration method, a computer device, and a storage medium. Based on a calibration reference object image and a first point cloud data set, the method determines a first external parameter matrix from a three-dimensional scanner to a first sensor. Then, based on the first external parameter matrix, the first point cloud data set, and a second point cloud data set, it determines a second external parameter matrix from the second sensor to the three-dimensional scanner. Finally, according to the first external parameter matrix and the second external parameter matrix, it determines a target external parameter matrix from the second sensor to the first sensor. The whole process does not involve extracting the features of the calibration reference object from a point cloud data set with low resolution (sparse point cloud data set), eliminating the influence of feature extraction errors on the calibration result, thereby improving the accuracy of the external parameters of multi-sensor joint calibration.

[0026] The multi-sensor joint calibration method provided by an embodiment of the present application can be applied to a computer device, which may include a terminal device, a self-mobile device, or a server. The terminal device may include a personal computer or a laptop computer, etc. The self-mobile device may include a sweeping robot or a lawn mowing robot. The server may be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.

[0027] The following will describe in detail some embodiments of the present invention with reference to the accompanying drawings. Without conflict, the following embodiments and the features in the embodiments may be combined with each other.

[0028] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of a multi-sensor joint calibration method provided by an embodiment of the present invention.

[0029] As Figure 1 shown, the multi-sensor joint calibration method includes steps S101 to S104.

[0030] Step S101, obtain a calibration reference object image, a first point cloud data set, and a second point cloud data set.

[0031] In this embodiment, the calibration reference object image is collected by the first sensor for a target scene provided with a calibration reference object. The first point cloud data set is obtained by the three-dimensional scanner scanning the target scene. The second point cloud data set is obtained by the second sensor scanning the target scene. The resolution of the first point cloud data set is higher than that of the second point cloud data set. Among them, the first sensor includes a vision sensor, and the second sensor includes a radar. The radar may include a lidar or a millimeter-wave radar.

[0032] In some embodiments, one or more calibration reference objects are set in the target scene. Among them, the calibration reference object may include a calibration board or a cooperative target. The calibration board may be a checkerboard calibration board, a Charuco calibration board, an AprilTag calibration board, etc. For example, when a vision sensor captures a target scene with an AprilTag calibration board, a calibration reference object image as shown in Figure 2 can be obtained, and when a 3D scanner scans a target scene with an AprilTag calibration board, a point cloud map corresponding to the second point cloud dataset as shown in Figure 3 can be obtained.

[0033] In some embodiments, obtaining the calibration reference object image, the first point cloud dataset, and the second point cloud dataset may include: controlling a first sensor to capture a target scene with a calibration reference object to obtain a calibration reference object image; controlling a 3D scanner to scan the target scene to obtain first point cloud data; controlling a second sensor to scan the target scene to obtain second point cloud data. In this embodiment, the first sensor, the second sensor, and the 3D scanner can be automatically controlled to collect corresponding data, reducing manual intervention, improving the efficiency of multi-sensor joint calibration of external parameters, and the high-precision 3D scanner can obtain high-resolution point cloud data, which can be reused multiple times in subsequent calculations without the need for frequent re-acquisition, improving the reusability of the calibration process.

[0034] Step S102: Determine a first external parameter matrix from the 3D scanner to the first sensor according to the calibration reference object image and the first point cloud dataset.

[0035] In this embodiment, multiple corner points can be extracted from the calibration reference object image and multiple corner points can be extracted from the first point cloud dataset, and then an iterative optimization algorithm is used to determine a first external parameter matrix from the 3D scanner to the first sensor based on the multiple corner points extracted from the calibration reference object image and the multiple corner points extracted from the first point cloud dataset.

[0036] In some embodiments, determining the first extrinsic matrix from the 3D scanner to the first sensor based on the calibration reference object image and the first point cloud dataset may include: extracting a plurality of first corner points from the calibration reference object image, where the first corner points may be feature points corresponding to the calibration reference object in the calibration reference object image, such as the corner points of the calibration board in the calibration reference object. When there are multiple calibration boards, their corner points correspond to the corner points of multiple calibration boards; and extracting a plurality of second corner points from the first point cloud dataset; matching the plurality of first corner points and the plurality of second corner points to obtain a plurality of matching corner point pairs, where a matching corner point pair includes a first corner point and a matching second corner point; adjusting the extrinsic matrix from the 3D scanner to the first sensor through an iterative optimization algorithm until the projection residuals corresponding to the plurality of matching corner point pairs are minimized, to obtain the first extrinsic matrix from the 3D scanner to the first sensor. By minimizing the residuals generated during the projection of the plurality of matching corner point pairs, the accuracy of the projection result is ensured. Specifically, the projection residual refers to the difference between the actual image point and the projected point when the 3D point is projected onto the 2D image plane through the extrinsic matrix. In this embodiment, since the resolution of the first point cloud dataset is relatively high, the accuracy of the plurality of second corner points extracted from the first point cloud dataset can be guaranteed. Thus, based on the plurality of corner points extracted from the calibration reference object image and the plurality of corner points extracted from the first point cloud dataset, the first extrinsic matrix from the 3D scanner to the first sensor can be accurately determined, improving the accuracy of the first extrinsic matrix.

[0037] In some embodiments, extracting a plurality of first corner points from the calibration reference object image and extracting a plurality of second corner points from the first point cloud dataset may include: extracting a plurality of first corner points from the calibration reference object image according to a preset corner point extraction algorithm; extracting the calibration reference object point cloud dataset from the first point cloud dataset, fitting the calibration reference object point cloud dataset to obtain the calibration reference object plane; projecting the reference object point cloud dataset onto the calibration reference object plane to obtain the projected point cloud dataset, and converting the projected point cloud dataset into a target image; obtaining the two-dimensional coordinates of a plurality of corner points from the target image based on the preset corner point extraction algorithm, mapping the two-dimensional coordinates of the plurality of corner points back to the local coordinate system where the calibration reference object plane is located to obtain a plurality of local three-dimensional coordinates; according to the transformation relationship between the local coordinate system and the global coordinate system where the first point cloud dataset is located, converting the plurality of local three-dimensional coordinates into a plurality of global three-dimensional coordinates, and determining the points corresponding to the plurality of global three-dimensional coordinates in the first point cloud dataset as the second corner points. Among them, the preset corner point extraction algorithm may include the Harris corner detection algorithm, the Shi-Tomasi corner detection algorithm, or the Apritag algorithm, etc.

[0038] In some embodiments, matching multiple first corner points and multiple second corner points to obtain multiple pairs of matching corner points may include: arranging the multiple first corner points based on their two-dimensional coordinates in the order from left to right and from top to bottom to obtain a list of first corner points; extracting a calibration reference object point cloud dataset from the first point cloud dataset, fitting the calibration reference object point cloud dataset to obtain a calibration reference object plane; projecting the multiple second corner points onto the calibration reference object plane to obtain the two-dimensional coordinates of the multiple second corner points, and performing principal component analysis on the two-dimensional coordinates of the multiple second corner points to obtain a first principal component direction and a second principal component direction; adjusting the first principal component direction to be consistent with the direction of the rows of the calibration reference object, and adjusting the second principal component direction to be consistent with the direction of the columns of the calibration reference object; taking the adjusted first principal component direction as the column direction and the adjusted second principal component direction as the row direction, and arranging the multiple second corner points based on their two-dimensional coordinates to obtain a list of second corner points; determining the index of each first corner point according to the list of first corner points, and determining the index of each second corner point according to the list of second corner points, and determining the first corner point and the second corner point with the same index as a pair of matching corner points. This embodiment can accurately establish multiple pairs of matching corner points according to the geometric relationship between the calibration reference object image and the first point cloud dataset, ensuring accurate calculation of the external parameter matrix from the 3D scanner to the first sensor.

[0039] In some embodiments, adjusting the external parameter matrix from the 3D scanner to the first sensor through an iterative optimization algorithm until the projection residuals corresponding to the multiple pairs of matching corner points are minimized to obtain the first external parameter matrix from the 3D scanner to the first sensor may include: determining an initial value of the external parameter matrix from the 3D scanner to the first sensor according to the multiple pairs of matching corner points; according to the internal parameter matrix of the first sensor and the current value of the external parameter matrix from the 3D scanner to the first sensor, projecting the second corner point in each pair of matching corner points onto the image coordinate system to obtain the projection point of the second corner point in each pair of matching corner points; determining the residual between the first corner point in each pair of matching corner points and the projection point corresponding to the matching second corner point, and determining the projection residuals corresponding to the multiple pairs of matching corner points according to the residual between the first corner point in each pair of matching corner points and the projection point corresponding to the matching second corner point; in response to the projection residuals not being the minimum, adjusting the value of the external parameter matrix from the 3D scanner to the first sensor, and returning to execute the step of projecting the second corner point in each pair of matching corner points onto the image coordinate system according to the internal parameter matrix of the first sensor and the current value of the external parameter matrix from the 3D scanner to the first sensor to obtain the projection point of the second corner point in each pair of matching corner points; in response to the projection residuals being the minimum, obtaining the first external parameter matrix from the 3D scanner to the first sensor. Among them, the iterative optimization algorithm may include the PnP algorithm.

[0040] Step S103: Determine the second extrinsic matrix from the second sensor to the 3D scanner according to the first extrinsic matrix, the first point cloud dataset, and the second point cloud dataset.

[0041] In this embodiment, the initial extrinsic matrix from the second sensor to the 3D scanner can be determined based on the first extrinsic matrix, and then the extrinsic matrix from the second sensor to the 3D scanner is adjusted through an iterative optimization algorithm with the initial extrinsic matrix from the second sensor to the 3D scanner as the initial value, so that the projection residual between the first point cloud dataset and the second point cloud dataset is minimized, and the second extrinsic matrix from the second sensor to the 3D scanner is obtained. Among them, the iterative optimization algorithm can include the Iterative Closest Point (ICP) algorithm. In this embodiment, by first calculating the high-precision first extrinsic matrix and then calculating the low-precision second extrinsic matrix, the gradual accumulation of errors in the system can be reduced, and the sensor fusion accuracy of the entire system can be improved.

[0042] In some embodiments, as Figure 4 shown, step S103 includes sub-steps S1031 to S1032.

[0043] Sub-step S1031: Determine the initial extrinsic matrix from the second sensor to the 3D scanner according to the first extrinsic matrix and the preset extrinsic matrix from the second sensor to the first sensor. The preset extrinsic matrix is related to the positional relationship between the second sensor and the first sensor.

[0044] In this embodiment, the positional relationship between the second sensor and the first sensor includes the relative distance and relative orientation between the second sensor and the first sensor. For example, both the first sensor and the second sensor are installed on the self-mobile device, and the positional relationship between the second sensor and the first sensor includes the relative distance and relative orientation between the installation position of the first sensor in the self-mobile device and the installation position of the second sensor in the self-mobile device. Among them, when determining the positional relationship between the second sensor and the first sensor, the preset extrinsic matrix from the second sensor to the first sensor can be accurately determined.

[0045] In some embodiments, determining the initial extrinsic matrix from the second sensor to the 3D scanner according to the first extrinsic matrix and the preset extrinsic matrix from the second sensor to the first sensor can include: multiplying the first extrinsic matrix by the preset extrinsic matrix from the second sensor to the first sensor to obtain the initial extrinsic matrix from the second sensor to the 3D scanner. For example, T fl_init = T fc * T cl_init , where T fl_init is the initial extrinsic matrix from the second sensor to the 3D scanner, T fc is the first extrinsic matrix, and T cl_initis the preset extrinsic parameter matrix from the second sensor to the first sensor.

[0046] Sub-step S1032: Using an iterative optimization algorithm, based on the initial extrinsic parameter matrix, the first point cloud dataset, and the second point cloud dataset, determine the second extrinsic parameter matrix from the second sensor to the 3D scanner.

[0047] In this embodiment, through the first extrinsic parameter matrix and the preset extrinsic parameter matrix from the second sensor to the first sensor, the initial extrinsic parameter matrix from the second sensor to the 3D scanner can be determined more accurately. In this way, when determining the second extrinsic parameter matrix from the second sensor to the 3D scanner through an iterative optimization algorithm with the initial extrinsic parameter matrix from the second sensor to the 3D scanner as the initial value, based on the first point cloud dataset and the second point cloud dataset, it is possible to avoid falling into a local optimum, further improving the accuracy of the second extrinsic parameter matrix from the second sensor to the 3D scanner. Thus, the accuracy of the target extrinsic parameter matrix from the second sensor to the first sensor determined based on the first extrinsic parameter matrix and the second extrinsic parameter matrix is further improved. In addition, by matching the second point cloud dataset with the first point cloud dataset, the first point cloud dataset is fully utilized. Compared with only using the point cloud data of the calibration reference object in the first point cloud dataset, the influence of noise on the calibration result is reduced, and the accuracy of the target extrinsic parameter matrix is further improved.

[0048] In some embodiments, using an iterative optimization algorithm to determine the second extrinsic parameter matrix from the second sensor to the 3D scanner based on the initial extrinsic parameter matrix, the first point cloud dataset, and the second point cloud dataset may include: According to the initial extrinsic parameter matrix from the second sensor to the 3D scanner, project each point in the second point cloud dataset into the coordinate system corresponding to the first point cloud dataset to obtain a projected point cloud dataset; determine the nearest neighbor points of each point in the projected point cloud dataset in the first point cloud dataset to obtain a plurality of matching point pairs; according to the plurality of matching point pairs, determine the projection residual between the first point cloud dataset and the second point cloud dataset; in response to the projection residual between the first point cloud dataset and the second point cloud dataset being greater than the preset projection residual threshold, determine an extrinsic parameter matrix increment according to the plurality of matching point pairs, update the initial extrinsic parameter matrix according to the extrinsic parameter matrix increment, and return to execute the step of projecting each point in the second point cloud dataset into the coordinate system corresponding to the first point cloud dataset according to the initial extrinsic parameter matrix from the second sensor to the 3D scanner to obtain a projected point cloud dataset until the projection residual between the first point cloud dataset and the second point cloud dataset is less than or equal to the preset projection residual threshold or the update times of the initial extrinsic parameter matrix reach the preset iteration times, thereby obtaining the second extrinsic parameter matrix from the second sensor to the 3D scanner.

[0049] In some embodiments, determining the projection residual between the first point cloud data set and the second point cloud data set according to multiple pairs of matching points may include: determining the Euclidean distance of each pair of matching points, accumulating the Euclidean distances of each pair of matching points, and then dividing the sum by the total number of pairs of matching points to obtain the projection residual between the first point cloud data set and the second point cloud data set. Alternatively, determining the square value of the Euclidean distance of each pair of matching points; accumulating the square values of the Euclidean distances of each pair of matching points, and then dividing the sum by the total number of pairs of matching points to obtain the projection residual between the first point cloud data set and the second point cloud data set.

[0050] In some embodiments, as Figure 5 shown, sub-step S1032 may include:

[0051] Sub-step S10321, updating the attitude angles in the initial extrinsic parameter matrix according to the first point cloud data set and the second point cloud data set.

[0052] In this embodiment, the displacement accuracy in the initial extrinsic parameter matrix from the second sensor to the 3D scanner is relatively high, while the accuracy of the attitude angles is relatively low. Therefore, updating the attitude angles in the initial extrinsic parameter matrix based on the first point cloud data set and the second point cloud data set can improve the accuracy of the attitude angles in the initial extrinsic parameter matrix, so that the accuracy of the second extrinsic parameter matrix from the second sensor to the 3D scanner can be further improved when using the iterative optimization algorithm. Among them, updating the attitude angles in the initial extrinsic parameter matrix may include updating the roll angle and the pitch angle in the initial extrinsic parameter matrix and / or updating the heading angle in the initial extrinsic parameter matrix.

[0053] In some embodiments, updating the attitude angles in the initial extrinsic parameter matrix according to the first point cloud data set and the second point cloud data set may include: projecting each point in the second point cloud data set into the coordinate system corresponding to the first point cloud data set according to the initial extrinsic parameter matrix to obtain a first projected point cloud data set; extracting a ground point cloud data set from the first projected point cloud data set; and updating the roll angle and the pitch angle in the initial extrinsic parameter matrix according to the ground point cloud data set and the first point cloud data set. In this embodiment, by updating the roll angle and the pitch angle in the initial extrinsic parameter matrix, the accuracy of the roll angle and the pitch angle in the initial extrinsic parameter matrix is improved, thereby improving the accuracy of the initial extrinsic parameter matrix.

[0054] In some embodiments, updating the roll angle and pitch angle in the initial extrinsic matrix based on the ground point cloud dataset and the first point cloud dataset may include: performing plane fitting on the ground point cloud dataset to obtain a first plane and determining the normal vector of the first plane; performing plane fitting on the first point cloud dataset to obtain a second plane and determining the normal vector of the second plane; updating the roll angle and pitch angle in the initial extrinsic matrix according to the angle between the normal vector of the first plane and the normal vector of the second plane. Wherein, the projected normal vector obtained by transforming the normal vector of the first plane based on the initial extrinsic matrix with the updated roll angle and pitch angle is aligned with the normal vector of the second plane. The plane fitting algorithm in the embodiments of the present invention may include the least squares method or the random sample consensus (RANSAC) algorithm, etc. In this embodiment, the angle between the normal vector of the plane obtained by fitting the ground point cloud dataset and the normal vector of the plane obtained by fitting the first point cloud dataset can accurately update the roll angle and pitch angle in the initial extrinsic matrix so that the two normal vectors can be aligned.

[0055] In some embodiments, extracting the ground point cloud dataset from the first projected point cloud dataset may include: randomly selecting at least three candidate ground points from the first projected point cloud dataset and performing plane fitting on the at least three candidate ground points to obtain a candidate ground plane, where the height of the candidate ground points is less than or equal to a preset height threshold; determining the first distance from each point in the first projected point cloud dataset to the candidate ground plane and determining the points with the first distance less than the first distance threshold as the inliers of the candidate ground plane; repeating the steps of randomly selecting at least three candidate ground points from the first projected point cloud dataset, performing plane fitting on the at least three candidate ground points to obtain a candidate ground plane, determining the first distance from each point in the first projected point cloud dataset to the candidate ground plane, and determining the points with the first distance less than the first distance threshold as the inliers of the candidate ground plane for n times to obtain n + 1 candidate ground planes and the inliers of each candidate ground plane, where n is an integer greater than or equal to 2; determining the candidate ground plane with the largest number of inliers as the target ground plane and extracting the points located in the target ground plane from the first projected point cloud dataset to obtain the ground point cloud dataset. This embodiment can accurately extract the ground point cloud dataset from the first projected point cloud dataset. Wherein, the first distance threshold can be set based on the actual situation, and the embodiments of the present invention do not make specific limitations on this.

[0056] In some embodiments, randomly selecting at least three candidate ground points from the first projected point cloud dataset may include: screening out points with a height less than or equal to a preset height threshold from the first projected point cloud dataset to obtain a candidate ground point set; determining the number of neighboring points of each candidate ground point in the candidate ground point set, and removing candidate ground points with a number of neighboring points less than or equal to a preset number threshold from the candidate ground point set to update the candidate ground point set; randomly selecting at least three candidate ground points from the updated candidate ground point set. Wherein, the number of neighboring points of a candidate ground point refers to the number of points with a distance less than a preset neighborhood distance from the candidate ground point. The preset height threshold can be determined according to the installation height of the second sensor. The preset number threshold and the preset distance threshold can be set based on actual situations, and the embodiments of the present invention do not make specific limitations in this regard.

[0057] In some embodiments, determining the candidate ground plane with the largest number of inliers as the target ground plane includes: determining the candidate ground plane with the largest number of inliers as the plane to be grown, and using the inliers of the plane to be grown as seed points; searching for neighborhood points of the seed points in the first projected point cloud dataset, and determining the second distance from the neighborhood points to the plane to be grown; when there is at least one neighborhood point with a second distance to the plane to be grown less than a second distance threshold, using at least one neighborhood point as new inliers of the plane to be grown to update the plane to be grown; using the new inliers as new seed points, and returning to execute the step of searching for neighborhood points of the seed points in the first projected point cloud dataset and determining the second distance from the neighborhood points to the plane to be grown; when the second distances of all neighborhood points to the plane to be grown are greater than or equal to the second distance threshold, stopping updating the plane to be grown, and determining the latest plane to be grown as the target ground plane. Wherein, the neighborhood points of the seed points are points with a distance less than a preset neighborhood distance from the seed points. The second distance threshold can be set based on actual situations, and the embodiments of the present invention do not make specific limitations in this regard. This embodiment further expands the candidate ground plane with the largest number of inliers, so as to obtain a more accurate target ground plane.

[0058] In some embodiments, updating the attitude angles in the initial extrinsic parameter matrix according to the first point cloud dataset and the second point cloud dataset may include: adjusting the heading angle in the initial extrinsic parameter matrix multiple times to obtain multiple candidate extrinsic parameter matrices; according to the candidate extrinsic parameter matrices, projecting each point in the second point cloud dataset onto the coordinate system corresponding to the first point cloud dataset to obtain a second projected point cloud dataset; determining the matching error between the second projected point cloud dataset and the first point cloud dataset; and updating the heading angle in the initial extrinsic parameter matrix according to the heading angle in the candidate extrinsic parameter matrix corresponding to the minimum matching error among the multiple candidate extrinsic parameter matrices. In this embodiment, by updating the roll angle and pitch angle in the initial extrinsic parameter matrix, the accuracy of the heading angle in the initial extrinsic parameter matrix is improved, thereby improving the accuracy of the initial extrinsic parameter matrix.

[0059] In some embodiments, adjusting the heading angle in the initial extrinsic parameter matrix multiple times to obtain multiple candidate extrinsic parameter matrices may include: starting from the heading angle in the initial extrinsic parameter matrix, adjusting the heading angle in the initial extrinsic parameter matrix within a preset heading angle range with a preset step size to obtain multiple candidate extrinsic parameter matrices. Wherein, the preset heading angle range and the preset step size can be set based on actual situations, and the embodiments of the present invention do not make specific limitations in this regard. For example, the preset heading angle range is [-4°, 4°], [-5°, 5°] or [-6°, 6°], etc., and the preset step size is 0.05, 0.1, 0.15 or 0.2, etc.

[0060] In some embodiments, determining the matching error between the second projected point cloud dataset and the first point cloud dataset may include: determining the nearest neighbor points of each point in the second projected point cloud dataset in the first point cloud dataset, and determining the Euclidean distance between each point in the second projected point cloud dataset and the corresponding nearest neighbor point; determining the matching error between the second projected point cloud dataset and the first point cloud dataset according to the Euclidean distance between each point in the second projected point cloud dataset and the corresponding nearest neighbor point. Wherein, when determining the nearest neighbor points, a KD tree can be constructed using the first point cloud dataset, and then the KD tree can be used to search for the nearest neighbor points to improve the search speed. In this embodiment, the matching error between the second projected point cloud dataset and the first point cloud dataset can be accurately determined through the Euclidean distance between each point in the second projected point cloud dataset and the corresponding nearest neighbor point.

[0061] In some embodiments, determining the matching error between the second projected point cloud dataset and the first point cloud dataset according to the Euclidean distance between each point in the second projected point cloud dataset and its corresponding nearest neighbor point may include: calculating the average Euclidean distance according to the Euclidean distance between each point in the second projected point cloud dataset and its corresponding nearest neighbor point, and determining the average Euclidean distance as the matching error between the second projected point cloud dataset and the first point cloud dataset. Alternatively, determining the square of the Euclidean distance between each point in the second projected point cloud dataset and its corresponding nearest neighbor point, accumulating the squares of the Euclidean distances between each point in the second projected point cloud dataset and its corresponding nearest neighbor point, and then dividing the sum by the total number of points in the second projected point cloud dataset to obtain the matching error between the second projected point cloud dataset and the first point cloud dataset.

[0062] In some embodiments, updating the attitude angles in the initial external parameter matrix according to the first point cloud dataset and the second point cloud dataset may include: projecting each point in the second point cloud dataset into the coordinate system corresponding to the first point cloud dataset according to the initial external parameter matrix to obtain a first projected point cloud dataset; extracting a ground point cloud dataset from the first projected point cloud dataset; updating the roll angle and pitch angle in the initial external parameter matrix according to the ground point cloud dataset and the first point cloud dataset; and making multiple adjustments to the heading angle in the initial external parameter matrix to obtain multiple candidate external parameter matrices; projecting each point in the second point cloud dataset into the coordinate system corresponding to the first point cloud dataset according to the candidate external parameter matrices to obtain a second projected point cloud dataset; determining the matching error between the second projected point cloud dataset and the first point cloud dataset; and updating the heading angle in the initial external parameter matrix according to the heading angle in the candidate external parameter matrix corresponding to the minimum matching error among the multiple candidate external parameter matrices. In this embodiment, by updating the roll angle, pitch angle, and heading angle in the initial external parameter matrix, the accuracy of the roll angle, pitch angle, and heading angle in the initial external parameter matrix is improved, thereby further improving the accuracy of the initial external parameter matrix.

[0063] Sub-step S10322: Using an iterative optimization algorithm, based on the updated initial external parameter matrix, the first point cloud dataset, and the second point cloud dataset, determine the second external parameter matrix from the second sensor to the 3D scanner.

[0064] In this embodiment, by updating the attitude angles in the initial extrinsic matrix, the accuracy of the initial extrinsic matrix can be further improved. In this way, through the iterative optimization algorithm with the updated initial extrinsic matrix from the second sensor to the 3D scanner as the initial value, when determining the second extrinsic matrix from the second sensor to the 3D scanner based on the first point cloud dataset and the second point cloud dataset, it is possible to further avoid falling into local optimality, further improve the accuracy of the second extrinsic matrix from the second sensor to the 3D scanner, and thus further improve the accuracy of the target extrinsic matrix from the second sensor to the first sensor determined based on the first extrinsic matrix and the second extrinsic matrix.

[0065] In some embodiments, using the iterative optimization algorithm to determine the second extrinsic matrix from the second sensor to the 3D scanner based on the updated initial extrinsic matrix, the first point cloud dataset, and the second point cloud dataset may include: determining the updated initial extrinsic matrix as the extrinsic matrix to be optimized; according to the extrinsic matrix to be optimized, projecting each point in the second point cloud dataset into the coordinate system corresponding to the first point cloud dataset to obtain a projected point cloud dataset; determining the nearest neighbor points of each point in the projected point cloud dataset in the first point cloud dataset to obtain a plurality of matching point pairs; according to the plurality of matching point pairs, determining the projection residual between the first point cloud dataset and the second point cloud dataset; in response to the projection residual between the first point cloud dataset and the second point cloud dataset being greater than a preset projection residual threshold, determining an extrinsic matrix increment according to the plurality of matching point pairs, updating the extrinsic matrix to be optimized according to the extrinsic matrix increment, and returning to execute the step of projecting each point in the second point cloud dataset into the coordinate system corresponding to the first point cloud dataset according to the extrinsic matrix to be optimized to obtain a projected point cloud dataset until the projection residual between the first point cloud dataset and the second point cloud dataset is less than or equal to the preset projection residual threshold or the update times of the initial extrinsic matrix reach the preset iteration times, thereby obtaining the second extrinsic matrix from the second sensor to the 3D scanner.

[0066] Step S104, determine the target extrinsic matrix from the second sensor to the first sensor according to the first extrinsic matrix and the second extrinsic matrix.

[0067] In this embodiment, based on the calibration reference object image and the first point cloud dataset, the first extrinsic matrix from the 3D scanner to the first sensor is determined, then based on the first extrinsic matrix, the first point cloud dataset, and the second point cloud dataset, the second extrinsic matrix from the second sensor to the 3D scanner is determined, and finally, according to the first extrinsic matrix and the second extrinsic matrix, the target extrinsic matrix from the second sensor to the first sensor is determined. The whole process does not involve extracting the features of the calibration reference object from the point cloud dataset with low resolution (sparse point cloud dataset), eliminating the influence of feature extraction error on the calibration result, and thus improving the accuracy of the extrinsic calibration of multi-sensor joint calibration.

[0068] In some embodiments, determining the target extrinsic matrix from the second sensor to the first sensor based on the first extrinsic matrix and the second extrinsic matrix includes: determining the inverse matrix of the first extrinsic matrix; performing a multiplication operation on the inverse matrix of the first extrinsic matrix and the second extrinsic matrix to obtain the target extrinsic matrix from the second sensor to the first sensor. For example, T cl = T fc -1 * T fl , where T cl is the target extrinsic matrix from the second sensor to the first sensor, T fc -1 is the inverse matrix of the first extrinsic matrix, and T fl is the second extrinsic matrix. In this embodiment, neither the process of determining the first extrinsic matrix nor the second extrinsic matrix involves extracting the features of the calibration reference object from the point cloud dataset with low resolution (sparse point cloud dataset), eliminating the influence of feature extraction errors on the first extrinsic matrix and the second extrinsic matrix, and ensuring the accuracy of the first extrinsic matrix and the second extrinsic matrix. Further, in the process of determining the second extrinsic matrix, the first point cloud dataset is fully utilized. Compared with only using the point cloud data of the calibration reference object in the first point cloud dataset, the influence of noise on the calibration result is reduced, and further, the accuracy of the second extrinsic matrix is improved. Furthermore, based on the high-precision first extrinsic matrix and the preset extrinsic matrix related to the positional relationship between the second sensor and the first sensor, a rough initial extrinsic matrix from the second sensor to the 3D scanner is estimated, and then the rough initial extrinsic matrix is further updated to obtain an accurate initial extrinsic matrix, so that when determining the second extrinsic matrix, it is possible to avoid falling into a local optimum, and further improve the accuracy of the second extrinsic matrix. In this way, based on the accurate first extrinsic matrix and the second extrinsic matrix, the target extrinsic matrix from the second sensor to the first sensor can be accurately determined, improving the accuracy of the extrinsic calibration of multi-sensor joint calibration.

[0069] Please refer to Figure 6 , Figure 6 which is a schematic block diagram of the structure of a computer device provided by an embodiment of the present invention.

[0070] As Figure 6 shown, the computer device 100 includes a processor 101 and a memory 102. The processor 101 and the memory 102 are connected through a bus 103, and this bus is, for example, an I2C (Inter - integrated Circuit) bus.

[0071] Specifically, the processor 101 is used to provide computing and control capabilities to support the operation of the entire computer device. The processor 101 can be a Central Processing Unit (CPU), or it can also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0072] Specifically, the memory 102 can be a Flash chip, Read-Only Memory (ROM), magnetic disk, optical disc, USB flash drive, or mobile hard disk, etc.

[0073] Those skilled in the art can understand that Figure 6 the structure shown in is only a block diagram of some structures related to the solution of the embodiment of the present invention, and does not constitute a limitation on the computer device to which the solution of the embodiment of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0074] Among them, the processor is used to run the computer program stored in the memory and implement any one of the multi-sensor joint calibration methods provided by the embodiments of the present invention when executing the computer program.

[0075] In one embodiment, the processor is used to run the computer program stored in the memory and implement the following steps when executing the computer program:

[0076] Obtain a calibration reference object image, a first point cloud data set, and a second point cloud data set. The calibration reference object image is collected by a first sensor for a target scene provided with the calibration reference object. The first point cloud data set is obtained by a 3D scanner scanning the target scene. The second point cloud data set is obtained by a second sensor scanning the target scene. The resolution of the first point cloud data set is higher than that of the second point cloud data set;

[0077] Determine a first external parameter matrix from the 3D scanner to the first sensor according to the calibration reference object image and the first point cloud data set;

[0078] Determine a second extrinsic matrix of the second sensor relative to the 3D scanner according to the first extrinsic matrix, the first point cloud dataset, and the second point cloud dataset;

[0079] Determine a target extrinsic matrix of the second sensor relative to the first sensor according to the first extrinsic matrix and the second extrinsic matrix.

[0080] In some embodiments, when the processor implements determining the second extrinsic matrix of the second sensor relative to the 3D scanner according to the first extrinsic matrix, the first point cloud dataset, and the second point cloud dataset, it is configured to:

[0081] Determine an initial extrinsic matrix of the second sensor relative to the 3D scanner according to the first extrinsic matrix and a preset extrinsic matrix of the second sensor relative to the first sensor, where the preset extrinsic matrix is related to the positional relationship between the second sensor and the first sensor;

[0082] Use an iterative optimization algorithm to determine the second extrinsic matrix of the second sensor relative to the 3D scanner based on the initial extrinsic matrix, the first point cloud dataset, and the second point cloud dataset.

[0083] In some embodiments, when the processor implements using an iterative optimization algorithm to determine the second extrinsic matrix of the second sensor relative to the 3D scanner based on the initial extrinsic matrix, the first point cloud dataset, and the second point cloud dataset, it is configured to:

[0084] Update the attitude angles in the initial extrinsic matrix according to the first point cloud dataset and the second point cloud dataset;

[0085] Use an iterative optimization algorithm to determine the second extrinsic matrix of the second sensor relative to the 3D scanner based on the updated initial extrinsic matrix, the first point cloud dataset, and the second point cloud dataset.

[0086] In some embodiments, when the processor implements updating the attitude angles in the initial extrinsic matrix according to the first point cloud dataset and the second point cloud dataset, it is configured to:

[0087] According to the initial extrinsic matrix, project each point in the second point cloud dataset onto the coordinate system corresponding to the first point cloud dataset to obtain a first projected point cloud dataset; extract a ground point cloud dataset from the first projected point cloud dataset; update the roll angle and pitch angle in the initial extrinsic matrix according to the ground point cloud dataset and the first point cloud dataset;

[0088] and / or

[0089] Adjust the heading angle in the initial extrinsic parameter matrix multiple times to obtain multiple candidate extrinsic parameter matrices; project each point in the second point cloud dataset into the coordinate system corresponding to the first point cloud dataset according to the candidate extrinsic parameter matrices to obtain a second projected point cloud dataset; determine the matching error between the second projected point cloud dataset and the first point cloud dataset; update the heading angle in the initial extrinsic parameter matrix according to the heading angle in the candidate extrinsic parameter matrix corresponding to the minimum matching error among the multiple candidate extrinsic parameter matrices.

[0090] In some embodiments, when the processor realizes updating the roll angle and pitch angle in the initial extrinsic parameter matrix according to the ground point cloud dataset and the first point cloud dataset, it is used to realize:

[0091] Perform plane fitting on the ground point cloud dataset to obtain a first plane and determine the normal vector of the first plane;

[0092] Perform plane fitting on the first point cloud dataset to obtain a second plane and determine the normal vector of the second plane;

[0093] Update the roll angle and pitch angle in the initial extrinsic parameter matrix according to the angle between the normal vector of the first plane and the normal vector of the second plane.

[0094] In some embodiments, when the processor realizes determining the matching error between the second projected point cloud dataset and the first point cloud dataset, it is used to realize:

[0095] Determine the nearest neighbor points of each point in the second projected point cloud dataset in the first point cloud dataset and determine the Euclidean distance between each point in the second projected point cloud dataset and the corresponding nearest neighbor point;

[0096] Determine the matching error according to the Euclidean distance between each point in the second projected point cloud dataset and the corresponding nearest neighbor point.

[0097] In some embodiments, when the processor realizes extracting the ground point cloud dataset from the first projected point cloud dataset, it is used to realize:

[0098] Randomly select at least three candidate ground points from the first projected point cloud dataset and perform plane fitting on the at least three candidate ground points to obtain a candidate ground plane, where the height of the candidate ground points is less than or equal to a preset height threshold;

[0099] Determine the first distance from each point in the first projected point cloud dataset to the candidate ground plane and determine the points with the first distance less than the first distance threshold as the inliers of the candidate ground plane;

[0100] Repeat the step of randomly selecting at least three candidate ground points from the first projected point cloud dataset, performing plane fitting on the at least three candidate ground points to obtain a candidate ground plane, determining a first distance from each point in the first projected point cloud dataset to the candidate ground plane, and determining points with a first distance less than a first distance threshold as inliers of the candidate ground plane for n times, to obtain n + 1 candidate ground planes and inliers of each candidate ground plane, where n is an integer greater than or equal to 2;

[0101] Determine the candidate ground plane with the largest number of inliers as the target ground plane, and extract points located within the target ground plane from the first projected point cloud dataset to obtain the ground point cloud dataset.

[0102] In some embodiments, when the processor implements determining the candidate ground plane with the largest number of inliers as the target ground plane, it is used to implement:

[0103] Determine the candidate ground plane with the largest number of inliers as the plane to be grown, and use the inliers of the plane to be grown as seed points;

[0104] Search for neighborhood points of the seed points in the first projected point cloud dataset, and determine a second distance from the neighborhood points to the plane to be grown;

[0105] When there is at least one neighborhood point with a second distance to the plane to be grown less than a second distance threshold, use the at least one neighborhood point as new inliers of the plane to be grown to update the plane to be grown;

[0106] Use the new inliers as new seed points, and return to execute the step of searching for neighborhood points of the seed points in the first projected point cloud dataset and determining a second distance from the neighborhood points to the plane to be grown;

[0107] When the second distances from all neighborhood points to the plane to be grown are greater than or equal to the second distance threshold, stop updating the plane to be grown, and determine the latest plane to be grown as the target ground plane.

[0108] In some embodiments, when the processor implements determining a first external parameter matrix of the 3D scanner to the first sensor according to the calibrated reference object image and the first point cloud dataset, it is used to implement:

[0109] Extract a plurality of first corner points from the calibrated reference object image and extract a plurality of second corner points from the first point cloud dataset;

[0110] Match the multiple first corner points and the multiple second corner points to obtain multiple pairs of matching corner points, where each pair of matching corner points includes one of the first corner points and one of the second corner points that matches it;

[0111] Adjust the external parameter matrix of the 3D scanner to the first sensor through an iterative optimization algorithm until the projection residuals corresponding to the multiple pairs of matching corner points are minimized, and obtain the first external parameter matrix of the 3D scanner to the first sensor.

[0112] Determine the first external parameter matrix of the 3D scanner to the first sensor based on the multiple pairs of matching corner points and the internal parameter matrix of the first sensor.

[0113] It should be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the computer device described above can refer to the corresponding process in the embodiment of the multi-sensor joint calibration method described above, and will not be repeated here.

[0114] The embodiment of the present invention also provides a storage medium for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement any multi-sensor joint calibration method provided in the specification of the embodiment of the present invention.

[0115] Among them, the storage medium can be the internal storage unit of the computer device described in the foregoing embodiment, such as the hard disk or memory of the computer device. The storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the computer device.

[0116] Those of ordinary skill in the art will understand that all or some of the steps in the methods disclosed above, and the functional modules / units in systems and devices, can be implemented as software, firmware, hardware, and their appropriate combinations. In a hardware embodiment, the division of functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be executed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transitory medium). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical disk storage, magnetic cartridges, tapes, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, a communication medium typically contains computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0117] It should be understood that the term "and / or" used in the specification and appended claims of the present invention refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations. It should be noted that in this article, the terms "include", "comprise", or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such a process, method, article, or system. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article, or system including that element.

[0118] The serial numbers of the embodiments of the present invention above are only for description and do not represent the superiority or inferiority of the embodiments. The above is only the specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A multi-sensor joint calibration method, characterized in that, Including: Obtain a calibration reference object image, a first point cloud dataset, and a second point cloud dataset. The calibration reference object image is acquired by a first sensor for a target scene provided with the calibration reference object. The first point cloud dataset is obtained by a 3D scanner scanning the target scene. The second point cloud dataset is obtained by a second sensor scanning the target scene. The resolution of the first point cloud dataset is higher than that of the second point cloud dataset; Determine a first extrinsic parameter matrix of the 3D scanner relative to the first sensor according to the calibration reference object image and the first point cloud dataset; Determine a second extrinsic parameter matrix of the second sensor relative to the 3D scanner according to the first extrinsic parameter matrix, the first point cloud dataset, and the second point cloud dataset; Determine a target extrinsic parameter matrix of the second sensor relative to the first sensor according to the first extrinsic parameter matrix and the second extrinsic parameter matrix.

2. The multi-sensor joint calibration method according to claim 1, wherein The step of determining the second extrinsic parameter matrix of the second sensor relative to the 3D scanner according to the first extrinsic parameter matrix, the first point cloud dataset, and the second point cloud dataset includes: Determine an initial extrinsic parameter matrix of the second sensor relative to the 3D scanner according to the first extrinsic parameter matrix and a preset extrinsic parameter matrix of the second sensor relative to the first sensor. The preset extrinsic parameter matrix is related to the positional relationship between the second sensor and the first sensor; Use an iterative optimization algorithm to determine the second extrinsic parameter matrix of the second sensor relative to the 3D scanner based on the initial extrinsic parameter matrix, the first point cloud dataset, and the second point cloud dataset.

3. The multi-sensor joint calibration method according to claim 2, wherein, The step of using an iterative optimization algorithm to determine the second extrinsic parameter matrix of the second sensor relative to the 3D scanner based on the initial extrinsic parameter matrix, the first point cloud dataset, and the second point cloud dataset includes: Update the attitude angles in the initial extrinsic parameter matrix according to the first point cloud dataset and the second point cloud dataset; Use an iterative optimization algorithm to determine the second extrinsic parameter matrix of the second sensor relative to the 3D scanner based on the updated initial extrinsic parameter matrix, the first point cloud dataset, and the second point cloud dataset.

4. The multi-sensor joint calibration method according to claim 3, characterized in that, The step of updating the attitude angles in the initial extrinsic parameter matrix according to the first point cloud dataset and the second point cloud dataset includes: According to the initial extrinsic parameter matrix, project each point in the second point cloud dataset onto the coordinate system corresponding to the first point cloud dataset to obtain a first projected point cloud dataset; extract a ground point cloud dataset from the first projected point cloud dataset; update the roll angle and pitch angle in the initial extrinsic parameter matrix according to the ground point cloud dataset and the first point cloud dataset; and / or Adjust the heading angle in the initial extrinsic parameter matrix multiple times to obtain multiple candidate extrinsic parameter matrices; project each point in the second point cloud dataset into the coordinate system corresponding to the first point cloud dataset according to the candidate extrinsic parameter matrices to obtain a second projected point cloud dataset; determine the matching error between the second projected point cloud dataset and the first point cloud dataset; update the heading angle in the initial extrinsic parameter matrix according to the heading angle in the candidate extrinsic parameter matrix corresponding to the smallest matching error among the multiple candidate extrinsic parameter matrices.

5. The multi-sensor joint calibration method according to claim 4, wherein The updating of the roll angle and pitch angle in the initial extrinsic parameter matrix according to the ground point cloud dataset and the first point cloud dataset includes: Perform plane fitting on the ground point cloud dataset to obtain a first plane and determine the normal vector of the first plane; Perform plane fitting on the first point cloud dataset to obtain a second plane and determine the normal vector of the second plane; Update the roll angle and pitch angle in the initial extrinsic parameter matrix according to the angle between the normal vector of the first plane and the normal vector of the second plane.

6. The multi-sensor joint calibration method according to claim 4, wherein The determining of the matching error between the second projected point cloud dataset and the first point cloud dataset includes: Determine the nearest neighbor point of each point in the second projected point cloud dataset in the first point cloud dataset, and determine the Euclidean distance between each point in the second projected point cloud dataset and the corresponding nearest neighbor point; Determine the matching error according to the Euclidean distance between each point in the second projected point cloud dataset and the corresponding nearest neighbor point.

7. The multi-sensor joint calibration method according to claim 4, wherein The extracting of the ground point cloud dataset from the first projected point cloud dataset includes: Randomly select at least three candidate ground points from the first projected point cloud dataset, and perform plane fitting on the at least three candidate ground points to obtain a candidate ground plane, where the height of the candidate ground points is less than or equal to a preset height threshold; Determine the first distance from each point in the first projected point cloud dataset to the candidate ground plane, and determine the points with the first distance less than the first distance threshold as the inliers of the candidate ground plane; Repeat the steps of randomly selecting at least three candidate ground points from the first projected point cloud dataset, performing plane fitting on the at least three candidate ground points to obtain a candidate ground plane, determining the first distance from each point in the first projected point cloud dataset to the candidate ground plane, and determining the points with the first distance less than the first distance threshold as the inliers of the candidate ground plane for n times to obtain n + 1 candidate ground planes and the inliers of each candidate ground plane, where n is an integer greater than or equal to 2; Determine the candidate ground plane with the largest number of inliers as the target ground plane, and extract the points located in the target ground plane from the first projected point cloud dataset to obtain the ground point cloud dataset.

8. The multi-sensor joint calibration method according to claim 7, wherein The determining of the candidate ground plane with the largest number of inliers as the target ground plane includes: Determine the candidate ground plane with the largest number of the inner points as the plane to be grown, and use the inner points of the plane to be grown as seed points; Search for the neighborhood points of the seed points in the first projected point cloud dataset, and determine the second distance from the neighborhood points to the plane to be grown; When there is at least one neighborhood point whose second distance to the plane to be grown is less than the second distance threshold, use the at least one neighborhood point as the new inner points of the plane to be grown to update the plane to be grown; Use the new inner points as new seed points, and return to execute the steps of searching for the neighborhood points of the seed points in the first projected point cloud dataset and determining the second distance from the neighborhood points to the plane to be grown; When the second distances from all the neighborhood points to the plane to be grown are greater than or equal to the second distance threshold, stop updating the plane to be grown, and determine the latest plane to be grown as the target ground plane.

9. The multi-sensor joint calibration method according to any one of claims 1-8, characterized in that The determining the first external parameter matrix from the three-dimensional scanner to the first sensor according to the calibrated reference object image and the first point cloud dataset includes: Extract a plurality of first corner points from the calibrated reference object image and extract a plurality of second corner points from the first point cloud dataset; Match the plurality of first corner points and the plurality of second corner points to obtain a plurality of matched corner point pairs, where each matched corner point pair includes one of the first corner points and a matched second corner point; Adjust the external parameter matrix from the three-dimensional scanner to the first sensor through an iterative optimization algorithm until the projection residuals corresponding to the plurality of matched corner point pairs are minimized, so as to obtain the first external parameter matrix from the three-dimensional scanner to the first sensor. Determine the first external parameter matrix from the three-dimensional scanner to the first sensor based on the plurality of matched corner point pairs and the internal parameter matrix of the first sensor.

10. The multi-sensor joint calibration method according to any one of claims 1-8, characterized in that, The determining the target external parameter matrix from the second sensor to the first sensor according to the first external parameter matrix and the second external parameter matrix includes: Determine the inverse matrix of the first external parameter matrix; Perform a multiplication operation on the inverse matrix of the first external parameter matrix and the second external parameter matrix to obtain the target external parameter matrix from the second sensor to the first sensor.

11. A computer device, characterized in that, The computer device includes a processor, a memory, a computer program stored on the memory and executable by the processor, and a data bus for realizing the connection communication between the processor and the memory. When the computer program is executed by the processor, the multi-sensor joint calibration method according to any one of claims 1 to 10 is realized.

12. A storage medium for computer-readable storage, characterized in that, The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to realize the multi-sensor joint calibration method according to any one of claims 1 to 10.

Images