Laser radar and camera target-level semantic fusion method and system based on inverse projection

Abstract

The invention relates to the technical field of robot perception and automatic driving, and discloses a laser radar and camera target-level semantic fusion method and system based on inverse projection, and the method comprises the steps: constructing a laser radar-camera calibration model based on environment arc features, extracting and optimizing the coordinates of the intersection point of the image arc features and the laser radar, and obtaining a target-level semantic fusion model; solving a projection matrix between the sensors; a multi-dimensional semantic segmentation algorithm based on inverse projection is designed, a segmentation region is obtained through inverse projection of a target bounding box, and unified segmentation of 2D / 3D laser point clouds is realized in combination with ground point cloud filtering, distance adaptive clustering, regional point cloud purification and regional growth; constructing a target-level semantic fusion framework, and integrating laser radar depth information and camera texture information; the method can improve the semantic segmentation precision (mIoU on a SemanticKITTI data set reaches 77.1% and exceeds 1.4% of the prior art) under the scene of lack of dense mark data, sparse point cloud or distance change, reduces background interference and calculation load, and is suitable for scenes of mobile robots, automatic driving and the like.

Description

Technical Field

[0001] This invention relates to the field of robot perception and autonomous driving technology, specifically to a target-level semantic fusion method and system based on inverse projection lidar and camera. Background Technology

[0002] In the fields of mobile robotics and autonomous driving, semantic fusion is one of the core technologies for improving environmental perception. LiDAR (Light Detection and Ranging) provides high-precision depth information but lacks texture features; cameras provide rich color and texture information, but depth measurement is easily affected by lighting conditions. Fusion of these two technologies can complement each other's shortcomings; however, existing technologies face the following key challenges:

[0003] Poor calibration accuracy and environmental adaptability: Traditional LiDAR-camera calibration relies on artificial targets such as checkerboard patterns and specific polygons, requiring data collection from multiple locations, which is cumbersome and prone to cumulative errors; targetless calibration methods mostly rely on indoor straight line features (such as walls and corners), and the accuracy drops sharply in industrial environments or outdoor scenes with sparse features, and cannot utilize curved environmental features (such as circular road signs and pipes).

[0004] Semantic segmentation is sensitive to distance and point cloud sparsity: In existing segmentation algorithms, traditional projection methods tend to introduce a large number of background points, resulting in low computational efficiency; neural network-based methods (such as PointPainting and Cylinder3D) require large-scale labeled data, and as the target distance increases and the point cloud becomes sparsity, the segmentation accuracy decreases significantly (e.g., when the distance exceeds 50m, mIoU drops to below 60%).

[0005] There is a lack of a unified multidimensional point cloud segmentation scheme: 2D LiDAR point clouds are only distributed on a single plane and lack contour features; 3D LiDAR point clouds are multiple layers of 2D point clouds superimposed. The data structures of the two are very different. Existing algorithms are mostly designed for a single dimension and cannot achieve unified semantic segmentation of 2D / 3D point clouds.

[0006] Insufficient computational efficiency and real-time performance: Deep learning-based semantic segmentation methods require a large amount of computational resources and a long processing time, making it difficult to meet the needs of application scenarios with high real-time requirements.

[0007] Therefore, there is an urgent need for a high-precision semantic fusion method that can adapt to complex environment calibration, reduce the impact of distance and point cloud sparsity, and support multi-dimensional point clouds. Summary of the Invention

[0008] This invention provides a target-level semantic fusion method and system for lidar and camera based on inverse projection, which helps to solve the problems mentioned in the background art.

[0009] This invention provides the following technical solution:

[0010] To achieve the above objectives, the present invention adopts the following technical solution: a target-level semantic fusion method for lidar and camera based on inverse projection, comprising:

[0011] Arc-feature-based LiDAR-camera calibration, inverse projection-based multidimensional point cloud semantic segmentation, and target-level semantic fusion;

[0012] Optional, LiDAR-camera calibration based on environmental arc characteristics, including:

[0013] By utilizing arc-shaped targets in the environment, including circular road signs and spheres, a spatial correspondence between the lidar and the camera is established. The specific steps are as follows:

[0014] Step S11: Image feature extraction:

[0015] Line feature extraction: The Contrast-Guided Line Detection (CGLSD) algorithm is used, guided by edge contrast, to enhance line segment continuity. After filtering noise, the equations of straight lines at the object edges are extracted. (Since many interfering straight lines appear when fitting pixel line segments in the image, a slope threshold can be used for filtering.) );

[0016] Arc feature extraction: An improved LSD algorithm is used to detect arc support segments from the denoised grayscale image and filter out the number of support points. And span angle The ellipse is taken as the target ellipse, and its mathematical expression is:

[0017] ;

[0018] Corresponding matrix form:

[0019]

[0020] in These represent the pixel coordinates of the pixel in the image. The coefficients are those obtained by transforming the equation of an ellipse into a quadratic form.

[0021] Step S12: Optimization of LiDAR intersection coordinates:

[0022] When a lidar scans a target plane, it intersects with the target boundary. , However, it is affected by angular resolution. Due to limitations, the actual detection point is the nearest point. ,in This represents the angular resolution multiple of the laser beam closest to the edge. This is the distance value returned by the laser beam;

[0023] To reduce angular error, adjustments are made based on the following rules:

[0024] If adjacent points All hit the target, and If the hit misses, the actual edge angle is at... ,Will Adjusted to ;

[0025] like Missed, , If it hits, then adjust to ;

[0026] Adjusted angle error constraint is within Improve coordinate accuracy;

[0027] Step S13: Solving for the projection matrix:

[0028] The spatial mapping relationship between the lidar and the camera satisfies the perspective projection model:

[0029] ;

[0030] Among them Original projection matrix , It is a 3×3 rotation matrix. It is a 3×1 translation vector; This is the camera intrinsic parameter matrix, obtained through camera calibration. These are the homogeneous coordinates of the lidar; These are the pixel coordinates of the image; For camera depth;

[0031] Due to 2D lidar points The mapping simplifies to a 2D homography transformation, taking... The first, second, and fourth columns form the projection matrix. ,satisfy:

[0032] ;

[0033] The solution process consists of two steps, including:

[0034] Linear solution: Based on point-line constraints, if the straight line satisfy Then, the 8-DOF system can be solved by SVD decomposition. Initial value;

[0035] Nonlinear optimization: with elliptic constraints To achieve the objective, the Levenberg-Marquardt algorithm is used to minimize the error:

[0036] ;

[0037] in The number of data sets must be greater than 4 to ensure sufficient constraints, ultimately yielding a high-precision projection matrix. .

[0038] Optionally, semantic segmentation of multidimensional laser point clouds based on inverse projection includes:

[0039] For unified segmentation of 2D / 3D point clouds, the target area is reduced through inverse projection and the point cloud is purified using multiple modules. The specific steps are as follows:

[0040] Step S21: Inverse projection segmentation:

[0041] Using the projection matrix of step S13 Inversely project the target bounding box in the camera image onto the lidar coordinate system:

[0042] For 2D point clouds: the two vertical sides of the bounding box are back-projected into two straight lines in the lidar coordinate system, i.e., two back-projection lines, and the target point cloud is located between the two straight lines.

[0043] For 3D point clouds: the bounding box back projection is a visual frustum region formed by the superposition of multiple 2D back projection regions, and the target point cloud is located inside the frustum.

[0044] The bounding box of the target in the camera image is obtained by segmentation using a YOLO-V10 instance with a confidence level ≥50%;

[0045] This process can filter out more than 70% of background points, significantly reducing the computational load in the subsequent calculations.

[0046] The mapping relationship of inverse projection satisfies ,in For the center projection mapping from the camera's optical center to the target point, This represents the truncation mapping of the projection lines on the laser scanning plane. The target point in the camera coordinate system This is the corresponding point in the lidar coordinate system;

[0047] Step S22: Ground point cloud filtering:

[0048] For 3D point clouds, a Simple Morphological Filter (SMRF) is used to remove ground points:

[0049] The point cloud is layered by height, with a layer thickness of 0.1m.

[0050] Morphological opening operations were performed on each layer using a disk structural element with a radius of 1m.

[0051] Non-ground points with a height difference ≥ 0.2m from the filtered ground are retained to obtain a refined point cloud;

[0052] 2D point clouds have no ground interference, so the point cloud from step S21 is directly retained;

[0053] Step S23: Distance-adaptive clustering:

[0054] The DBSCAN algorithm has been improved based on the distance between the target and the lidar. Adjusting the cluster radius Solving the problem of sparse point clouds:

[0055] LiDAR angular resolution Derive the linear relationship between cluster radius and distance:

[0056] ;

[0057] Where K is a constant;

[0058] Set reference distance Time reference radius The actual radius satisfies ;

[0059] For 2D point clouds, based on the distance between the laser point and the center of the lidar... Point-by-point adjustment ;

[0060] For 3D point clouds, adjust according to the distance to cluster centers. The initial clustering is obtained. ;

[0061] Step S24: Regional point cloud purification:

[0062] Two strategies are used to filter residual noise:

[0063] Similarity point removal: Calculate the cosine similarity between clusters based on the point cloud normal vectors. If the similarity is >50%, remove small clusters with fewer than 50 points. If the cluster sizes are equal and there is an inclusion relationship, remove clusters with a proportion of <30%.

[0064] Enlarged bounding box deletion: Enlarge the target bounding box by 10%, re-inverse project and cluster, and remove the clusters that are newly added after enlargement and whose number of points increases by more than 20% (most of which are background noise).

[0065] Step S25: Region Growth

[0066] Restore the target point cloud that was truncated by the camera's field of view:

[0067] Use the largest cluster obtained in step S24 as the seed;

[0068] Traverse the global point cloud, if the candidate point and the seed point have Euclidean distance If so, it is classified into the seed cluster;

[0069] Repeat step S25 until no new points are added, and the complete target point cloud is obtained.

[0070] Optional, target-level semantic fusion, including:

[0071] The semantic segmentation point cloud from step S25 is combined with the target semantic information (such as "car" and "pedestrian" labels) from the camera image through a projection matrix. The association establishes a one-to-one correspondence between "LiDAR point cloud - camera target - semantic label" and outputs a unified target-level fusion result that includes depth, texture and semantics.

[0072] The present invention has the following beneficial effects:

[0073] 1. Strong calibration adaptability: Utilizing arc features eliminates the need for artificial targets and maintains a calibration error of <0.5 pixels even in feature-sparse environments (such as tunnels and warehouses), which is superior to traditional line feature methods (error >1.2 pixels).

[0074] 2. High segmentation accuracy: The mIoU reaches 77.1% on the SemanticKITTI dataset, which is 1.4% higher than the existing EPMF algorithm, and the mIoU is still above 80% when the distance is 50-60m (existing algorithms <65%).

[0075] 3. Excellent multidimensional compatibility: It uniformly supports 2D / 3D laser point clouds without modifying the algorithm framework and adapts to different sensor configurations;

[0076] 4. Excellent real-time performance: No neural network training process, single frame processing time <0.1s (CPU: i5-9300H), meeting the real-time requirements of mobile robots;

[0077] 5. Strong distance robustness: The mIoU exceeds 80% in all distance segments from 0 to 60m, solving the problem of significant decrease in segmentation accuracy caused by increasing distance in existing technologies;

[0078] 6. High computational efficiency: By filtering more than 70% of background points through inverse projection segmentation, the subsequent processing load is greatly reduced, and the system operating efficiency is improved. Attached Figure Description

[0079] Figure 1 This describes the specific steps involved in a target-level semantic fusion method for LiDAR and camera based on inverse projection.

[0080] Figure 2 This is a schematic diagram of the lidar scanning of the present invention, showing the intersection of the laser beam and the target plane. and testing points Positional relationships;

[0081] Figure 3 : Coordinate system and camera imaging model diagram, showing the imaging process of the target on the image plane;

[0082] Figure 4 The diagram shows the inverse projection area. The second row is the inverse projection line formed by projecting the semantic object bounding box onto the 2D laser scanning plane. The third row is the visual frustum formed by projecting the semantic object bounding box onto the 3D laser scanning space. Detailed Implementation

[0083] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0084] Example 1, refer to Figure 1 A target-level semantic fusion method based on inverse projection lidar and camera includes the following steps:

[0085] S1. LiDAR-camera calibration based on environmental arc characteristics, specifically including:

[0086] In a specific implementation scenario, the lidar-camera calibration process of the present invention is as follows:

[0087] Step S11: Image feature extraction:

[0088] In an outdoor scene containing circular road signs, the CGLSD algorithm is used to process camera images, enhance edge contrast and extract line segments, and filter vertical edge lines by slope thresholding to ensure the accuracy of subsequent calibration.

[0089] Simultaneously, the ASLD algorithm was used to detect elliptical contours in the image, and target ellipses with 45 (≥30) support points and a span angle of 120° (≥90°) were selected. The equation of the ellipse is as follows: ;

[0090] The corresponding matrix form is:

[0091] ;

[0092] Step S12: Optimization of LiDAR intersection coordinates:

[0093] Reference Figure 2 A diagram illustrating how a lidar scans space and hits a target.

[0094] LiDAR angular resolution When scanning the target plane, points were detected. Adjacent points All hit the target, and Missed. According to the rules, the actual edge angle is adjusted to... The angle error decreased from 0.48° to 0.24°;

[0095] Step S13: Solving for the projection matrix:

[0096] Reference Figure 3 , Coordinate system and camera imaging model diagram.

[0097] Camera intrinsic parameter matrix It is known that:

[0098] ;

[0099] The initial projection matrix is ​​constructed by establishing point-line constraints based on the linear target contour equation. The projection matrix is ​​obtained by solving the SVD decomposition and then using the Levenberg-Marquardt algorithm for nonlinear optimization of the elliptical target contour equation. for:

[0100] ;

[0101] S2. Semantic segmentation of multidimensional laser point clouds based on inverse projection, specifically including:

[0102] After calibration, perform multidimensional laser point cloud semantic segmentation:

[0103] Step S21: Inverse projection segmentation:

[0104] Reference Figure 4 The diagram shows the inverse projection area. The second row is the inverse projection line formed by projecting the semantic object bounding box onto the 2D laser scanning plane. The third row is the visual frustum formed by projecting the semantic object bounding box onto the 3D laser scanning space.

[0105] A car target was detected in the camera image using YOLO-v10, with the bounding box coordinates as follows: ;

[0106] Using projection matrix By inversely projecting the bounding box onto the lidar coordinate system, for a 2D lidar point cloud, we obtain the equations of two inverse projection lines:

[0107] ;

[0108] ;

[0109] The target point cloud is located in the region between two straight lines.

[0110] Step S22: Ground point cloud filtering:

[0111] For 3D laser point clouds, the point cloud is layered with a height of 0.1m, and morphological opening operations are performed using a disk structure element with a radius of 1m. Points with a height difference of ≥0.2m from the filtered ground are retained to obtain a refined point cloud.

[0112] Step S23: Distance-adaptive clustering:

[0113] LiDAR angular resolution Calculated Set the reference distance , reference radius For a target point 30m away from the lidar, the cluster radius is calculated. .

[0114] Step S24: Regional point cloud purification:

[0115] Calculate the cosine similarity between each cluster. It is found that the similarity between cluster C1 (150 points) and C2 (80 points) is 60% (>50%). C2 cluster with fewer points is removed. At the same time, the target bounding box is enlarged by 10% and then re-projected to remove the cluster with a 25% (>20%) increase in the number of new points.

[0116] Step S25: Region Growth

[0117] Using the largest cluster C3 (200 points) as the seed, traverse the global point cloud; if the Euclidean distance between the candidate point and the seed point is... And the angle between the normal vectors If the target point cloud is not found, it will be classified into the C3 cluster. After region growing, the target point cloud is expanded from 200 points to 280 points, successfully restoring the part that was truncated by the camera's field of view.

[0118] S3. Target-level semantic fusion, specifically including:

[0119] The segmented point cloud is then projected through a matrix. Associated with camera target labels, the output contains a fusion result including point cloud coordinates, depth, and semantic labels;

[0120] In this embodiment, the point cloud of the vehicle target was successfully associated with the semantic label of "vehicle", thus achieving target-level semantic fusion.

[0121] Example 2: A system for implementing the target-level semantic fusion method of LiDAR and camera based on inverse projection, comprising:

[0122] Sensor module: Includes a 2D LiDAR (Hesai Pandar XT-32) and an industrial camera (Baslerac A2040-90um) for acquiring environmental point cloud and image data;

[0123] Feature extraction module: Employs the CGLSD algorithm and an improved LSD algorithm to extract line and arc features from the image;

[0124] Calibration module: Implements the coordinate optimization and projection matrix solution functions in steps S12-S13;

[0125] Semantic segmentation module: performs inverse projection segmentation, ground filtering, distance adaptive clustering, region cleanup and region growing operations in steps S21-S25;

[0126] Fusion module: Associates the segmented point cloud with the camera target semantic information and outputs a unified target-level semantic fusion result.

[0127] The system hardware configuration is: Intel i5-9300H processor, 8GB RAM, and the software environment is a mixed programming environment of MATLAB R2023a and C++.

[0128] In actual testing, the system processed one frame of data in 0.08 seconds, meeting the real-time requirement of 12.5fps; the mIoU on the SemanticKITTI dataset reached 77.1%, and the mIoU of each distance segment exceeded 80%, verifying the effectiveness of the invention.

[0129] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0130] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A target-level semantic fusion method for lidar and camera based on inverse projection, characterized in that, include: S1. LiDAR-Camera Calibration Based on Environmental Arc Features: S11. Extract line features and arc features from the image. The line features are obtained using the Contrast Guided Line Segment Detection (CGLSD) algorithm, and the arc features are extracted from the denoised grayscale image using the Arc Supported Line Segment Detection (ASLD) algorithm. S12. Optimize the target intersection point coordinates in the lidar coordinate system: based on the lidar angular resolution. Adjust the polar coordinates of the nearest point on the edge to ,in This represents the angular resolution multiple of the laser beam closest to the edge. The distance returned by the laser beam has a constraint angle error within... Within the range; S13. Constructing and solving the projection matrix: Based on the point-line constraint, the initial projection matrix is ​​solved using linear least squares and singular value decomposition (SVD). Then, the Levenberg-Marquardt algorithm is used for nonlinear optimization to obtain the projection matrix between the LiDAR and the camera. ; S2. Semantic segmentation of multidimensional laser point clouds based on inverse projection: S21. Inverse projection segmentation: Using the projection matrix obtained in step S13 The target bounding box in the camera image is inversely projected onto the lidar coordinate system to obtain an approximate region of the target point cloud; S22. Ground point cloud filtering: If the input is a 3D point cloud, a simple morphological filter (SMRF) is used to remove the ground point cloud to obtain a refined point cloud; if it is a 2D point cloud, the approximate region point cloud from step S21 is directly retained. S23. Distance-Adaptive Clustering: Based on the distance between LiDAR and the target Adaptive radius The DBSCAN algorithm, in which satisfy , is a constant and ; S24. Regional point cloud purification: Filter background noise through similar point deletion (removing small clusters with similarity exceeding 50%) and enlarged bounding box deletion (removing clusters with more than 5% of the same category of laser points after enlarging the bounding box); S25. Region growing: Using the largest point cloud cluster obtained in step S24 as a seed, expand and grow in the global point cloud to restore the target point cloud that was truncated by the camera's field of view. S3. Target-level semantic fusion: Associate the semantic segmentation point cloud obtained in step S2 with the target semantic information of the camera image, and output a unified target-level semantic fusion result.

2. The target-level semantic fusion method of lidar and camera based on inverse projection according to claim 1, characterized in that, In step S11, the actual environmental target corresponding to the arc feature is a circular road sign or a sphere, which is represented as an ellipse in the image. Its equation is obtained through a matrix. express: ； quadratic satisfy ,in These represent the pixel coordinates of the pixel in the image.

3. The target-level semantic fusion method for lidar and camera based on inverse projection according to claim 1, characterized in that, In step S13, the projection matrix The construction satisfies: ； in , , ; It is a 3×3 rotation matrix. It is a 3×1 translation vector. For the camera intrinsic parameter matrix, This represents the depth value of a point in the camera coordinate system. Image pixel coordinates, These are the coordinates of a point in the lidar coordinate system. For the first matrix List.

4. The target-level semantic fusion method of lidar and camera based on inverse projection according to claim 1, characterized in that, In step S13, the objective function for nonlinear optimization is: ； in The number of data sets, For the first The elliptical matrix corresponding to the set of data. The coordinates of the intersection point of the lidar. This represents the calibration matrix of the lidar and camera, and its transpose.

5. The target-level semantic fusion method of lidar and camera based on inverse projection according to claim 1, characterized in that, In step S21, the mapping relationship of the inverse projection satisfies ; in For the center projection mapping from the camera's optical center to the target point, This represents the truncation mapping of the projection lines on the laser scanning plane. The target point in the camera coordinate system This is the corresponding point in the lidar coordinate system.

6. The target-level semantic fusion method of lidar and camera based on inverse projection according to claim 1, characterized in that, In step S23, for the 2D point cloud: based on the distance from the center of the lidar to each point... Select Adaptive For 3D point clouds: select directly based on the distance between points. .

7. The target-level semantic fusion method for lidar and camera based on inverse projection according to claim 1, characterized in that, In step S24, the rules for deleting similar points are as follows: traverse all clusters, if the similarity between clusters exceeds 50%, remove the cluster with fewer points; if there is an inclusion relationship between clusters, remove the containing cluster with a smaller proportion.

8. The target-level semantic fusion method of lidar and camera based on inverse projection according to claim 1, characterized in that, In step S25, the neighborhood judgment criterion for region growing is: if the Euclidean distance between the candidate point and the seed point is less than that in step S23... If so, the candidate points will be assigned to the target cluster.

9. A system for implementing the aforementioned target-level semantic fusion method based on inverse projection lidar and camera, characterized in that, include: The processor and the program stored in processor-readable storage medium; When the processor executes the program, it implements the steps of the method according to any one of claims 1-8, and the system further includes: Sensor module: including LiDAR (2D / 3D) and camera, used to acquire environmental point cloud and image data; Feature extraction module: used to perform line and arc feature extraction in step S11; Calibration module: used to perform coordinate optimization and projection matrix solving in steps S12-S13; Semantic segmentation module: used to perform segmentation and purification in steps S21-S25; Fusion module: Used to perform target-level semantic information association and output in step S3.

Images