Semantic constraint-based laser radar and camera automatic external parameter calibration method

Abstract

The invention provides a semantic constraint-based laser radar and camera automatic external parameter calibration method, and relates to the technical field of laser radars, and the method comprises the following steps: S1, carrying out the image segmentation of a camera image through a large model, and obtaining the pedestrian mask information; s2, segmenting the radar point cloud, and endowing each point category sequence number to form a series of point cloud target clusters; s3, matching the pedestrian mask information with the point cloud target cluster; and S4, semi-global constraint online optimization is carried out on the successfully matched pedestrian targets, the final optimization target is that the number of the 3D pedestrian projection points falling on the pedestrian mask is the largest, and if the number of the 3D pedestrian projection points falling on the pedestrian mask is the largest, external reference is preserved and updated, otherwise, the optimization result is abandoned. Compared with some traditional methods, the method is higher in constraint capability and better in optimization effect. The method can be online and automatic, is high in robustness, is good in calibration effect, and can meet the requirements of complex and variable scenes.

Description

Technical Field

[0001] The present invention relates to the field of laser radar technology, and in particular provides a method for automatic extrinsic calibration of laser radar and camera based on semantic constraints. Background Art

[0002] In recent years, multi-sensor fusion technology has been widely used in unmanned driving, smart home, industrial automation and other fields. Multi-sensor fusion is the key to improving the robustness and accuracy of perception systems. LiDAR and cameras are usually important components of multi-sensor systems. Calibration can ensure that they are accurately aligned in the spatial coordinate system, thereby achieving information fusion and complementarity.

[0003] Although traditional LiDAR and camera calibration algorithms have made a lot of progress, they still have some limitations. For example, although the target-based calibration method can achieve high accuracy, it usually requires manual intervention, so it can only be calibrated offline and has a low degree of automation. The calibration method based on feature information can achieve a high degree of automation through scene feature extraction and matching, but the general accuracy and robustness are not high, and it depends on the scene geometric features. Usually, the algorithm fails due to the lack of specific geometric structures in the new scene. The calibration method based on mutual information maximization can achieve a certain degree of noise resistance by associating the image grayscale value with the point cloud intensity value information, but it is easily affected by lighting and has low accuracy. The motion-based calibration method models a series of sensor movement sequences as a homogeneous transformation equation to inversely solve the external parameters. The advantage is that there is no need for a common view area, but it is easily affected by noise and the accuracy depends on environmental feature information. The calibration method based on deep learning can achieve a high degree of automation, but the current accuracy is not high and the generalization ability is poor.

[0004] Therefore, traditional calibration algorithms are usually limited in their applicable scenarios, difficult to adapt to complex and changeable calibration environments, and cannot take into account performance indicators such as accuracy, robustness, and scenario versatility, and cannot meet the application requirements of complex scenarios. Summary of the invention

[0005] The purpose of this application is to solve the problem that the traditional method is limited in application scenarios and the calibration results are inaccurate, and to provide a calibration method that obtains better calibration results under given initial external parameters.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: a method for automatic extrinsic calibration of laser radar and camera based on semantic constraints, comprising the following steps:

[0007] S1. Use the large model to segment the camera image and obtain pedestrian mask information;

[0008] S2, segment the radar point cloud and assign a category number to each point to form a series of point cloud target clusters;

[0009] S3, matching the pedestrian mask information obtained by S1 with the point cloud target cluster formed by S2. If the match is successful, proceed to S4; if the match is unsuccessful, the data is invalid.

[0010] S4. Perform semi-global constrained online optimization on the successfully matched pedestrian targets. The ultimate goal of the optimization is to maximize the number of 3D pedestrian projection points falling on the pedestrian mask. The external parameters are saved and updated, otherwise the optimization results are discarded.

[0011] The method for automatic extrinsic calibration of laser radar and camera based on semantic constraints provided by the present invention also has the following technical features: S1 comprises:

[0012] S1.1: Use the MobileSAMv2 lightweight large model to segment multiple frames of camera images, and obtain a series of target masks for each frame;

[0013] S1.2: Use the YOLOv5 target detection algorithm to identify dynamic targets in multiple frames of images, and output the dynamic target information in the format of image name, target category, confidence, pixel coordinates of the upper left vertex of the 2D box, and width and height;

[0014] S1.3: Use the targets with the semantic category of person in the target detection results to calculate the IOU with a series of masks of the corresponding image segmentation one by one. When the IOU value is greater than 0.6, the corresponding unique pedestrian mask is matched, thereby filtering out other static background masks with ambiguous segmentation.

[0015] The semantic constraint-based automatic extrinsic parameter calibration method for lidar and camera provided by the present invention also has the following technical features: the series of target masks include a dynamic target mask for the foreground and a static target mask for the background; the dynamic target mask and the static target mask both include the pixel area of ​​the target, the pixel coordinates of the upper left vertex of the 2D box, and the width and height.

[0016] The semantic constraint-based automatic external parameter calibration method for lidar and camera provided by the present invention also has the following technical features: S2 includes: using the RANSAC algorithm to extract points in the point cloud to represent the ground and background; performing Euclidean clustering on the remaining points in the point cloud to form a series of point cloud target clusters.

[0017] The method for automatic extrinsic calibration of laser radar and camera based on semantic constraints provided by the present invention also has the following technical features: S3 includes:

[0018] S3.1: By measuring the translation distance and rotation angle of the laser radar and camera, the initial external parameters are obtained, and the point cloud target cluster is projected onto the corresponding image using the initial external parameters;

[0019] S3.2: count the point cloud cluster category numbers contained in the pixel area of ​​the pedestrian target in the camera image;

[0020] S3.3: Calculate the pixel area and 2D box information of the point cloud cluster counted in S3.2 falling on the image;

[0021] S3.4: Select a 3D point cloud pedestrian category cluster with the highest geometric similarity to the pedestrian target from multiple point cloud clusters, and the 3D point cloud pedestrian category cluster is the 3D point cloud pedestrian category cluster corresponding to the pedestrian target.

[0022] The semantic constraint-based automatic extrinsic parameter calibration method for laser radar and camera provided by the present invention also has the following technical features, wherein the geometric similarity includes the following: the pixel area of ​​the pedestrian in the camera image is similar to the pixel area occupied by the 3D pedestrian cluster projected onto the image; the 2D frame area of ​​the pedestrian in the camera image is similar to the 2D frame area projected onto the image by the 3D pedestrian cluster; the 2D frame aspect ratio of the pedestrian in the camera image is similar to the 2D frame aspect ratio of the 3D pedestrian cluster projected onto the image.

[0023] The method for automatic extrinsic calibration of laser radar and camera based on semantic constraints provided by the present invention also has the following technical features: S4 includes:

[0024] S4.1: The initial external parameters are obtained by measuring the translation distance and rotation angle of the laser radar and camera, and the sum of the number of projection points of all successfully matched 3D point cloud pedestrian categories in multiple sets of data that fall on the corresponding image pedestrian mask is counted;

[0025] S4.2: Fine-tune the external parameters within the fine-tuning range given by the initial external parameters;

[0026] S4.3: Use the new external parameters fine-tuned in S4.2 to project multiple groups of point cloud clusters into the corresponding image again, and count the sum of the number of points of the 3D pedestrian target cluster falling on the corresponding pedestrian mask. If the sum of the number of points of the 3D pedestrian cluster falling on the corresponding pedestrian mask is greater than the sum of the number of initial statistical points, the current external parameters are saved and updated, otherwise they are discarded;

[0027] S4.4: Repeat S4.2-S4.3, use multi-threading technology to fully traverse and search for new external parameters, continuously save and update the external parameters with a larger sum of the number of points of the 3D pedestrian cluster falling on the corresponding pedestrian mask, and finally output the external parameters that maximize the number of 3D pedestrian projection points falling on the pedestrian mask, and give the final visualization projection effect.

[0028] The semantic constraint-based automatic external parameter calibration method for lidar and camera provided by the present invention also has the following technical features: the fine-tuning range allows the roll angle, yaw angle, and pitch angle to fluctuate within plus or minus 5° respectively, and the translation vector (x, y, z) coordinates to fluctuate within plus or minus 0.5 meters.

[0029] Beneficial Effects

[0030] The automatic external parameter calibration method of the lidar and camera based on semantic constraints proposed by the present invention uses the MobileSAMv2 large model for image segmentation, and utilizes the zero-sample migration capability of the large model in new scenes to improve the scene universality of the calibration scheme. Since the large model itself lacks semantic information, pedestrian semantic information is introduced through the YOLOv5 target detection algorithm, and the pedestrian mask is matched and screened, and then the 2D and 3D pedestrian semantics are matched. The high-level semantic matching is more robust than the low-level feature point matching. Finally, by performing semi-global constrained online optimization on the successfully matched pedestrian targets, the specific optimization goal is to maximize the number of 3D pedestrian projection points falling on the pedestrian mask. This optimization method has stronger constraints and better optimization effects than some traditional methods. The present invention can be realized online and automatically, has high robustness, and has good calibration effects, and can meet the needs of complex and changeable scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 A principle framework diagram of the automated calibration method provided by an embodiment of the present invention;

[0033] Figure 2 A visualization diagram of the image segmentation effect of the MobileSAMv2 lightweight large model provided in an embodiment of the present invention;

[0034] Figure 3 A schematic diagram of the effect of YOLOv5 image target detection provided by an embodiment of the present invention;

[0035] Figure 4 A schematic diagram of the visualization projection effect of the point cloud target cluster on the image using the initial external parameters provided by an embodiment of the present invention;

[0036] Figure 5 Schematic diagram of geometric similarity matching between a pedestrian mask in an image and a pedestrian target cluster in a point cloud in an embodiment of the present invention;

[0037] Figure 6 Schematic diagram for comparing ideal external parameter optimization results and non-ideal optimization results in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The present application is further described in detail below in conjunction with the accompanying drawings and examples, but it should be noted that these implementation methods are not limitations of the present application, and equivalent changes or substitutions in functions, methods, or structures made by ordinary technicians in the field based on these implementation methods are all within the scope of protection of the present application.

[0039] In the description of the embodiments of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the creation of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the creation of the present application.

[0040] In addition, the terms "first", "second", "third", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the invention of this application, unless otherwise specified, the meaning of "plurality" is two or more.

[0041] The terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention of this application can be understood by specific circumstances.

[0042] like Figure 1-6 As shown, the embodiment of the present application provides a method for automatic extrinsic calibration of a laser radar and a camera based on semantic constraints, comprising the following steps:

[0043] S1. Use the large model to segment the camera image and obtain pedestrian mask information;

[0044] S2, segment the radar point cloud and assign a category number to each point to form a series of point cloud target clusters;

[0045] S3, matching the pedestrian mask information obtained by S1 with the point cloud target cluster formed by S2. If the match is successful, proceed to S4; if the match is unsuccessful, the data is invalid.

[0046] S4. Perform semi-global constrained online optimization on the successfully matched pedestrian targets. The ultimate goal of the optimization is to maximize the number of 3D pedestrian projection points falling on the pedestrian mask. The external parameters are saved and updated, otherwise the optimization results are discarded.

[0047] In the above embodiment, the laser radar coordinate system T Lidar 、Camera coordinate system T Camera and pixel coordinate system T Pixel , where the pixel coordinate system T Pixel is a two-dimensional coordinate system, and the others are three-dimensional coordinate systems.

[0048] In an embodiment of the present invention, the transformation from the laser radar coordinate system to the camera coordinate system is This is the parameter that the external parameter calibration method in the present invention ultimately requires. The transformation parameter required is a 4×4 matrix, including a 3×3 rotation matrix and a three-dimensional translation vector, where the unit of the rotation matrix is ​​radian and the unit of the translation vector is meter.

[0049] In some embodiments, the S1 includes:

[0050] S1.1: Use the MobileSAMv2 lightweight large model to segment multiple frames of camera images, and obtain a series of target masks for each frame;

[0051] S1.2: Use the YOLOv5 target detection algorithm to identify dynamic targets in multiple frames of images, and output the dynamic target information in the format of image name, target category, confidence, pixel coordinates of the upper left vertex of the 2D box, and width and height;

[0052] S1.3: Use the targets with the semantic category of person in the target detection results to calculate the IOU with a series of masks of the corresponding image segmentation one by one. When the IOU value is greater than 0.6, the corresponding unique pedestrian mask is matched, thereby filtering out other static background masks with ambiguous segmentation.

[0053] In the above embodiment, target detection algorithms such as YOLOv5 can be used to identify dynamic targets such as pedestrians and vehicles in the scene. Pedestrians are considered to be optimized rather than vehicles because pedestrians are usually slower, and the time synchronization accuracy requirements for the laser radar and camera are relatively loose. In addition, whether calibrated indoors or outdoors, data containing pedestrians is easy to obtain, so that the calibration scheme has almost no restrictions on the scene and is flexible and easy to implement. In addition, the size of a person is smaller than that of a vehicle, and the optimization of each pixel area of ​​the image is more fine-grained. In addition, after the corresponding pedestrian mask is matched by calculating the IOU, other useless masks need to be removed, and the corresponding mask information in the database needs to be updated and saved.

[0054] In some embodiments, the series of object masks include a dynamic object mask of the foreground and a static object mask of the background, and the dynamic object mask and the static object mask both include the pixel area of ​​the object, the pixel coordinates of the upper left vertex of the 2D box, and the width and height.

[0055] The large model has a better effect on foreground target segmentation, but the segmentation effect of some static backgrounds is sometimes ambiguous. In order not to affect the external parameter calibration effect, the background mask is filtered out through YOLOv5. In addition, the large model has zero-sample migration capability and can adapt to new external parameter calibration environments without additional training.

[0056] In some embodiments, S2 includes: extracting points in the point cloud for representing the ground and the background using a RANSAC algorithm; and performing Euclidean clustering on the remaining points in the point cloud to form a series of point cloud target clusters.

[0057] In some embodiments, point cloud segmentation assigns a category number to each point to form a series of point cloud target clusters, including the following steps:

[0058] 1) Use the RANSAC algorithm to extract larger planes in the point cloud. In one embodiment of the present invention, the plane fitting adopts the RANSAC (random sampling consensus) algorithm, and the obtained plane equation is Ax+By+Cz+D=0. In addition to considering the position of the point for fitting, the normal direction of the point is also calculated to provide stricter geometric constraints to ensure that the plane model fits more accurately. It should be noted that fitting the plane in the point cloud can help identify and eliminate local plane structures in the data, because large-area planes such as the ground or background that may exist in the point cloud are usually not the target of concern, which helps to reduce the computational complexity of subsequent Euclidean clustering and improve the effect of Euclidean clustering.

[0059] 2) Perform Euclidean clustering on the remaining points in the point cloud to form a series of point cloud target clusters. The point cloud cluster set is denoted as S = {s0, s1, s2, ..., s i ,...,s n-1}, where s i Represents the point cloud cluster with category number i, and there are n point cloud clusters in total. The algorithm implementation process of Euclidean clustering is roughly as follows: a) Initialize all points and mark them as "unvisited"; b) Randomly select a point from the unvisited points as a seed point, find all the unvisited points closest to the seed point, and calculate the Euclidean distance between them and the seed point; c) Classify the points whose distance from the seed point is less than the threshold into the same cluster, and mark these points as "visited"; d) For the newly added seed points, use them as seed points, repeat the above steps, and continue to expand the cluster until no more points can be added; e) Finally, all points have been visited, and each point is assigned to a separate cluster, thus completing the segmentation of the point cloud. It should be noted that in order to speed up the algorithm implementation process of Euclidean clustering, a KD-Tree (K-dimensional tree) can be constructed to accelerate the search for the nearest neighbor of the seed point, thereby improving the efficiency of the algorithm. In addition, the density of clustering can be controlled by adjusting the Euclidean distance threshold. A smaller threshold will produce more small clusters, while a larger threshold will result in fewer but larger clusters.

[0060] In some embodiments, S3 includes:

[0061] S3.1: By measuring the translation distance and rotation angle of the laser radar and camera, the initial external parameters are obtained. Using the initial external parameters, the point cloud target clusters are projected onto the corresponding image. The original image needs to be dedistorted first, and then all point cloud target clusters are projected onto the dedistorted image, such as Figure 4 As shown, it is obvious that the initial external parameters have some deviations. First, obtain the camera's intrinsic parameter matrix K and distortion coefficient D, where the intrinsic parameter matrix K is a 3×3 matrix in the form of:

[0062]

[0063] Among them, f x and f y Represents the horizontal and vertical focal length, c x and c y Represents pixel offset; the distortion coefficient D is a 5-dimensional vector in the form of:

[0064] D=[k1,k2,p1,p2,k3]

[0065] Among them, k1, k2, k3 are the first three coefficients of radial distortion, and p1, p2 are the first two coefficients of tangential distortion. Let (x, y) be the pixel coordinates of the original distorted image, and (x0, y0) be the pixel coordinates of the undistorted image. According to the principles of radial distortion and tangential distortion, we have the following formula:

[0066] Radial distortion:

[0067] Tangential distortion:

[0068] The three-dimensional coordinates of the point cloud are recorded as (X, Y, Z), and the points in the radar coordinate system are converted to the pixel coordinate system:

[0069]

[0070] Where W is The third coordinate of

[0071] S3.2: Count the point cloud cluster category numbers contained in the pixel area of ​​the pedestrian target in the camera image, traverse all point cloud target clusters, project them onto the dedistorted image, record the point cloud cluster category numbers that have overlapping pixel areas with the pedestrian target mask, and record the selected point cloud cluster set as R, and like Figure 5 As shown in the figure, the 3D point cloud clusters framed in the figure are all categories with overlapping pixel areas with the pedestrian mask, among which there are point cloud clusters corresponding to the pedestrian itself, and the rest are point cloud clusters corresponding to the background;

[0072] S3.3: Calculate the pixel area and 2D frame information of the point cloud clusters counted in S3.2 falling on the image. The pixel area is obtained by traversing the points of these point cloud clusters and counting the number of different pixels they occupy on the image. The 2D frame information is also obtained by traversing the points of the point cloud clusters and continuously updating and saving the maximum and minimum values ​​in the x-axis direction, as well as the maximum and minimum values ​​on the y-axis.

[0073] S3.4: Select a 3D point cloud pedestrian category cluster with the highest geometric similarity to the pedestrian target from multiple point cloud clusters, and the 3D point cloud pedestrian category cluster is the 3D point cloud pedestrian category cluster corresponding to the pedestrian target.

[0074] In some embodiments, the geometric similarity includes the following: the pixel area of ​​the pedestrian in the camera image is similar to the pixel area occupied by the 3D pedestrian cluster projected onto the image; the 2D frame area of ​​the pedestrian in the camera image is similar to the 2D frame area of ​​the 3D pedestrian cluster projected onto the image; the 2D frame aspect ratio of the pedestrian in the camera image is similar to the 2D frame aspect ratio of the 3D pedestrian cluster projected onto the image.

[0075] In some embodiments, the specific implementation process of geometric similarity judgment is as follows: First, respectively calculate Figure 5 The pixel area, 2D box area, and 2D box aspect ratio of the point cloud cluster framed in the middle are calculated, and then the pixel area, 2D box area, and 2D box aspect ratio of the pedestrian mask to be matched are calculated. Secondly, a geometric similarity evaluation function is constructed to calculate Figure 5 The similarity score of the framed point cloud cluster, the pixel area of ​​the point cloud cluster is Area p , the area of ​​the 2D box is 2D_Areap , the 2D frame aspect ratio is 2D_Ratio p , the pixel area of ​​the pedestrian mask to be matched is Area mask , the area of ​​the 2D box is 2D_Area mask , the 2D frame aspect ratio is 2D_Ratio mask , the geometric similarity score is Score, and the geometric similarity evaluation function is:

[0076]

[0077] Among them, w1, w2, and w3 are matching weight parameters, and the specific values ​​can be selected as needed within a certain range according to the actual scene. Figure 5 The geometric similarity score between the point cloud cluster framed in the figure and the pedestrian mask to be matched. The point cloud cluster with the highest score and exceeding a certain threshold is the pedestrian cluster corresponding to the pedestrian mask. Figure 5 The point cloud cluster framed by the pink box is the pedestrian cluster corresponding to the pedestrian mask. The matching between the 2D pedestrian mask and the 3D pedestrian cluster is now completed.

[0078] In some embodiments, the S4 includes:

[0079] S4.1: By measuring the translation distance and rotation angle of the laser radar and camera, the initial external parameters are obtained, and the sum of the number of projection points of all successfully matched 3D point cloud pedestrian categories in multiple groups of data falling on the corresponding image pedestrian mask is counted. It should be noted that Figure 5 The following is a schematic diagram of matching a single pedestrian mask with the corresponding pedestrian point cloud cluster. In order to build more geometric constraints and achieve more accurate external parameter optimization results, it is necessary to match 2D pedestrians with 3D pedestrian clusters for multiple frames of images and their time-synchronized point clouds. After the match is successful, the sum of the number of points where the 3D pedestrian cluster projection falls on the corresponding 2D pedestrian mask is counted when the initial external parameters are inaccurate. The formula is expressed as:

[0080] Point_num_sum=Point_num1+Point_num2+Point_num3+...+Point_num n

[0081] Among them, Point_num nis the number of projection points of the nth pedestrian cluster on the corresponding pedestrian mask, and Point_num_sum is the sum of the number of projection points of all successfully matched pedestrian clusters in multiple groups of data on the corresponding pedestrian mask. Obviously, when the external parameters are unbiased, the 3D pedestrian cluster should completely overlap with its corresponding 2D pedestrian mask after being projected on the image. At this time, the number of projection points of the 3D pedestrian cluster projected on the corresponding 2D pedestrian mask is the largest, then each item in the above formula can be maximized, so Point_num_sum is maximized, which is the goal of external parameter optimization;

[0082] S4.2: Fine-tune the external parameters within the fine-tuning range given by the initial external parameters;

[0083] S4.3: Use the new external parameters fine-tuned by S4.2 to project multiple groups of point cloud clusters into the corresponding image again, and count the sum of the number of points of the 3D pedestrian target cluster falling on the corresponding pedestrian mask. If the sum of the number of points of the 3D pedestrian cluster falling on the corresponding pedestrian mask is greater than the sum of the number of initial statistical points, the current external parameter is saved and updated, otherwise it is discarded; it should be noted that whether Point_num_sum becomes larger, but the external parameter is farther away from the true value, is also considered whether the external parameter optimization can be achieved by maximizing the constructed mathematical model Point_num_sum. Figure 6 As shown, the left sub-image is the ideal external parameter optimization result, and the projection points are evenly distributed in the entire pixel area of ​​the target mask. The right sub-image is the situation where the external parameter optimization result deteriorates, and the projection points are concentrated in a local area of ​​the target mask. However, the number of projection points of the pedestrian clusters in the left and right sub-images that fall on the corresponding pedestrian mask is the largest. However, this situation is only possible under the condition of a very small number of pedestrian constraints. When there are multiple sets of pedestrian data to establish more constraints, this situation does not exist. Therefore, external parameter optimization can be achieved by maximizing the constructed mathematical model Point_num_sum. As long as Point_num_sum is maximized, accurate external parameter optimization results can be achieved;

[0084] S4.4: Repeat S4.2-S4.3, use multi-threading technology to fully traverse and search for new external parameters, continuously save and update the external parameters with a larger sum of the number of points of the 3D pedestrian cluster falling on the corresponding pedestrian mask, and finally output the external parameters that maximize the number of 3D pedestrian projection points falling on the pedestrian mask, and give the final visualization projection effect.

[0085] In some embodiments, the fine-tuning range allows the roll angle, yaw angle, and pitch angle to fluctuate within plus or minus 5°, and the translation vector (x, y, z) coordinates to fluctuate within plus or minus 0.5 meters. In actual external parameter calibration application scenarios, some traditional calibration algorithms can often be used, such as extracting 2D and 3D corresponding feature point pairs to construct and solve the PnP (Perspective-n-Point) equation, or even manually fine-tuning the external parameters to obtain a relatively accurate but slightly biased initial external parameter result. Therefore, in general, fine-tuning the external parameters within a small angle and small displacement range can achieve the external parameter calibration function of this article. The specific method of fine-tuning the external parameters is to first decompose the external parameter transfer matrix into a rotation matrix and a translation vector, and then decompose the rotation matrix into three Euler angles in degrees, so that the Euler angles are randomly offset within plus or minus 5 degrees, and the coordinates of the three axes of the translation vector are randomly offset within plus or minus 0.5 meters. After completing the random offset, the offset Euler angles are converted back to the rotation matrix and combined with the offset translation vector into a new external parameter transfer matrix.

[0086] In the method provided in any of the above embodiments, a lightweight large model MobileSAMv2 is used to perform multi-frame image segmentation to obtain a series of dynamic and static mask images, and YOLOv5 target detection is used to identify pedestrian targets. The IOU is calculated with the mask image to match the corresponding pedestrian mask, and the static background mask with ambiguous segmentation effect is filtered out; secondly, the point cloud is segmented. The specific method adopts a planar segmentation method followed by Euclidean clustering to finally form a series of point cloud target clusters; then, the pedestrian mask in the image is matched with the corresponding pedestrian target cluster in the point cloud by 2D-3D semantic matching using geometric similarity. The specific matching method is to first count the 3D projection points contained in the pedestrian mask under the initial external parameters. Category number, and then use the geometric information features such as the area of ​​the pedestrian mask, the area of ​​the 2D box, the aspect ratio of the 2D box, etc. to match the geometric similarity with the area, 2D box area, and 2D box aspect ratio of the 3D point cluster projected on the image, so as to match the 3D category cluster that uniquely corresponds to the pedestrian mask; finally, semi-global constrained online optimization is performed on the successfully matched pedestrian targets. Specifically, the external parameters are fine-tuned in a small range, and the sum of the number of corresponding 3D pedestrian cluster projection points on all successfully matched pedestrian masks with different external parameters is continuously counted. Multi-threaded brute force search is used to save and update the external parameters with a larger sum of projection points. Finally, the external parameters that maximize the sum of corresponding projection points are output as the final optimization result. This new method of automated extrinsic parameter calibration of lidar and camera uses a lightweight large model MobileSAMv2 for image segmentation. Compared with traditional small segmentation models, the large model's zero-sample generalization capability makes this calibration method more scene-universal and better than traditional small segmentation models. Compared with the original SAM large model, MobileSAMv2 is more lightweight, greatly improving the segmentation efficiency while maintaining the same segmentation effect as the original SAM large model. Compared with low-level feature point matching, high-level semantics are used for 2D-3D matching, which is not easy to introduce accuracy errors, making the calibration method more resistant to interference and more fault-tolerant. Semi-global optimization is performed by maximizing the number of 3D pedestrian projection points falling on the pedestrian mask. This optimization method is guided by the ideal visualization projection effect, making the extrinsic parameter optimization results more accurate. This method not only has the above advantages from a technical perspective, but also has certain advantages in the simplicity and flexibility of actual operation. It does not require the use of special calibration plates to be placed in the scene in advance, and to be placed and fixed according to different postures, and does not require the preparation of a specific calibration environment. Outdoors, it only needs to collect some data of pedestrians or vehicles, and indoors, it only needs to have a person pass by once to obtain the data required for the calibration method in this article. Therefore, data acquisition is very convenient and flexible, and the external parameter optimization process can be highly automated and applicable to a wide range of scenarios. Therefore, this solution can achieve a more accurate and robust external parameter calibration effect.

[0087] The above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application. The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and variations can be made without departing from the technical principles of the present application, and these improvements and variations should also be regarded as the protection scope of the present application.

Claims

1. A method for automatic extrinsic calibration of laser radar and camera based on semantic constraints, characterized in that: The following steps are involved: S1. Use the large model to segment the camera image and obtain the pedestrian mask information; S2, segment the radar point cloud and assign a category number to each point to form a series of point cloud target clusters; S3, matching the pedestrian mask information obtained by S1 with the point cloud target cluster formed by S2. If the match is successful, proceed to S4; if the match is unsuccessful, the data is invalid. S4. Perform semi-global constrained online optimization on the successfully matched pedestrian targets. The ultimate goal of the optimization is to maximize the number of 3D pedestrian projection points falling on the pedestrian mask. The external parameters are saved and updated, otherwise the optimization results are discarded.

2. The method for automatic extrinsic calibration of laser radar and camera based on semantic constraints according to claim 1, characterized in that: The S1 includes: S1.1: Use the MobileSAMv2 lightweight large model to segment multiple frames of camera images, and obtain a series of target masks for each frame; S1.2: Use the YOLOv5 target detection algorithm to identify dynamic targets in multiple frames of images, and output the dynamic target information in the format of image name, target category, confidence, pixel coordinates of the upper left vertex of the 2D box, and width and height; S1.3: Use the targets with the semantic category of person in the target detection results to calculate the IOU with a series of masks of the corresponding image segmentation one by one. When the IOU value is greater than 0.6, the corresponding unique pedestrian mask is matched, thereby filtering out other static background masks with ambiguous segmentation.

3. The method for automatic extrinsic calibration of laser radar and camera based on semantic constraints according to claim 2, characterized in that: The series of object masks include a dynamic object mask of the foreground and a static object mask of the background, and the dynamic object mask and the static object mask both include the pixel area of ​​the object, the pixel coordinates of the upper left vertex of the 2D box, and the width and height.

4. The method for automatic extrinsic calibration of laser radar and camera based on semantic constraints according to claim 1, characterized in that: The S2 includes: using the RANSAC algorithm to extract points in the point cloud for representing the ground and the background; and performing Euclidean clustering on the remaining points in the point cloud to form a series of point cloud target clusters.

5. The method for automatic extrinsic calibration of laser radar and camera based on semantic constraints according to claim 1, characterized in that: The S3 includes: S3.1: By measuring the translation distance and rotation angle of the laser radar and camera, the initial external parameters are obtained, and the point cloud target cluster is projected onto the corresponding image using the initial external parameters; S3.2: count the point cloud cluster category numbers contained in the pixel area of ​​the pedestrian target in the camera image; S3.3: Calculate the pixel area and 2D box information of the point cloud cluster counted in S3.2 falling on the image; S3.4: Select a 3D point cloud pedestrian category cluster with the highest geometric similarity to the pedestrian target from multiple point cloud clusters, and the 3D point cloud pedestrian category cluster is the 3D point cloud pedestrian category cluster corresponding to the pedestrian target.

6. The method for automatic extrinsic calibration of laser radar and camera based on semantic constraints according to claim 5, characterized in that: The geometric similarity includes the following: the pixel area of ​​the pedestrian in the camera image is similar to the pixel area occupied by the 3D pedestrian cluster projected on the image; the 2D box area of ​​the pedestrian in the camera image is similar to the 2D box area of ​​the 3D pedestrian cluster projected on the image; the 2D box aspect ratio of the pedestrian in the camera image is similar to the 2D box aspect ratio of the 3D pedestrian cluster projected on the image.

7. The method for automatic extrinsic calibration of laser radar and camera based on semantic constraints according to claim 1, characterized in that: The S4 includes: S4.1: The initial external parameters are obtained by measuring the translation distance and rotation angle of the laser radar and camera, and the sum of the number of projection points of all successfully matched 3D point cloud pedestrian categories in multiple sets of data that fall on the corresponding image pedestrian mask is counted; S4.2: Fine-tune the external parameters within the fine-tuning range given by the initial external parameters; S4.3: Use the new external parameters fine-tuned in S4.2 to project multiple groups of point cloud clusters into the corresponding image again, and count the sum of the number of points of the 3D pedestrian target cluster falling on the corresponding pedestrian mask. If the sum of the number of points of the 3D pedestrian cluster falling on the corresponding pedestrian mask is greater than the sum of the number of initial statistical points, the current external parameters are saved and updated, otherwise they are discarded; S4.4: Repeat S4.2-S4.3, use multi-threading technology to fully traverse and search for new external parameters, continuously save and update the external parameters with a larger sum of the number of points of the 3D pedestrian cluster falling on the corresponding pedestrian mask, and finally output the external parameters that maximize the number of 3D pedestrian projection points falling on the pedestrian mask, and give the final visualization projection effect.

8. The method for automatic extrinsic calibration of laser radar and camera based on semantic constraints according to claim 7, characterized in that: The fine-tuning range allows the roll angle, yaw angle, and pitch angle to fluctuate within plus or minus 5 degrees respectively, and the translation vector (x, y, z) coordinates to fluctuate within plus or minus 0.5 meters.

Images