Multi-laser fusion point cloud three-dimensional target detection method based on two-dimensional vision enhancement

Abstract

The invention discloses a multi-laser fusion point cloud three-dimensional target detection method based on two-dimensional vision enhancement, and the method comprises the steps: carrying out the three-dimensional universal obstacle recognition of point clouds scanned by a single-line mechanical laser radar and a solid-state laser radar based on a local grid map and a light tracing method; calculating the terrain clearance of the three-dimensional universal obstacle identified by the solid-state laser radar; splicing the images shot by the plurality of cameras to form a two-dimensional panoramic image; performing target detection and semantic segmentation on the two-dimensional panoramic image by using a two-dimensional deep learning detection model and a segmentation model; and finally, projecting an identification result of the three-dimensional universal obstacle to the two-dimensional panoramic image, performing association matching with the two-dimensional detection frame, and performing semantic enhancement processing on the successfully matched three-dimensional universal obstacle to obtain independent three-dimensional targets corresponding to different semantic entities. The method can realize omnibearing perception and accurate measurement of the height of the obstacle, and is low in calculation complexity and strong in dynamic environment adaptability.

Description

A 3D target detection method based on multi-laser fusion point clouds enhanced by 2D vision Technical Field

[0001] This invention relates to the field of autonomous driving and environmental perception, and in particular to a three-dimensional target detection method based on multi-laser fusion point clouds enhanced by two-dimensional vision. Background Technology

[0002] With the rapid development of autonomous driving and intelligent transportation, intelligent unmanned vehicles, as key carriers for future urban travel and logistics distribution, directly determine driving safety and operational efficiency through their environmental perception capabilities. Existing unmanned vehicle perception solutions mainly face the following technical challenges: (1) Perception blind zone problem: Traditional single lidar solutions have a large blind zone in the near-ground area of ​​the vehicle, making it difficult to effectively detect low obstacles (such as road obstacles, leftover items, children's toys, etc.) and the feet of pedestrians suddenly crossing, which constitutes a safety hazard in low-speed and parking scenarios. (2) Insufficient recognition of aerial obstacles: Ordinary mechanical lidar has limited perception capabilities for suspended obstacles (such as traffic signs, temporary hanging objects, tree branches, overpasses and tunnel ancillary facilities), and the sparse point cloud leads to incomplete three-dimensional representation, affecting path planning and passability judgment. (3) Low efficiency of multi-sensor fusion: Existing lidar and vision fusion methods usually rely on the direct matching of three-dimensional point clouds and two-dimensional images, which has high computational complexity and is easily affected by calibration errors, lighting changes and occlusion, resulting in unstable fusion effects. (4) Weak adaptability to dynamic scenarios: In complex urban scenarios, traditional perception systems are not capable of real-time identification, tracking and behavior prediction of dynamic traffic participants (pedestrians, non-motorized vehicles and other vehicles), which makes it difficult to meet the needs of real-time decision-making and obstacle avoidance in highly dynamic environments.

[0003] Therefore, there is an urgent need to develop a method for comprehensive and high-precision three-dimensional environmental perception and target detection to improve the driving safety, decision-making reliability and overall intelligence level of intelligent unmanned vehicles in complex and dynamic traffic scenarios. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a three-dimensional target detection method based on two-dimensional vision enhancement and multi-laser fusion point cloud, which can improve the accuracy and real-time performance of three-dimensional target detection.

[0005] The objective of this invention is achieved through the following technical solution: a three-dimensional target detection method based on multi-laser fusion point clouds with two-dimensional visual enhancement, comprising: S1: using local grid maps and ray tracing methods, performing three-dimensional general obstacle identification on point clouds scanned by single-line mechanical lidar and solid-state lidar respectively, and calculating the ground clearance of the three-dimensional general obstacles identified by the solid-state lidar; S2: stitching together images captured by multiple cameras to form a two-dimensional panoramic image; using a two-dimensional deep learning detection model and segmentation model, performing target detection and semantic segmentation on the two-dimensional panoramic image to obtain two-dimensional detection results; S3: projecting the three-dimensional general obstacle identification results of S1 onto the two-dimensional panoramic image, and performing association matching with the two-dimensional detection boxes output by S2, performing semantic enhancement processing on the successfully matched three-dimensional general obstacles to obtain independent three-dimensional targets corresponding to different semantic entities.

[0006] Furthermore, based on local grid maps and ray tracing methods, the point cloud scanned by the single-line mechanical lidar is used for general obstacle identification, specifically including the following sub-steps: (1) Projecting the point cloud data of consecutive frames onto a two-dimensional grid map centered on the single-line mechanical lidar; (2) Each grid cell is marked as idle, unknown, or occupied by statistically analyzing the point cloud density and occupancy status; (3) Based on the ray tracing method, tracing back from the sensor origin to each occupied grid cell, analyzing the propagation path of the laser beam, and if the beam is terminated within the expected distance, the grid cell is confirmed as a real obstacle.

[0007] Furthermore, based on local grid maps and ray tracing methods, the point cloud scanned by the solid-state LiDAR is used for general obstacle recognition and ground clearance calculation. Specifically, this includes the following sub-steps: (a) Establishing a grid map centered on the solid-state LiDAR, projecting the point cloud scanned by the solid-state LiDAR onto the grid, sorting the points in the same grid by Z-value, considering the point with the smallest Z-value as the ground point, then using a region growing algorithm to calculate the ground expression, and then calculating the ground clearance of the remaining points, thus obtaining a binary classification of ground points and obstacle points; (b) Constructing a local grid map and using ray tracing technology to identify occupied grids, and then classifying the local grid... The map is binarized, and then morphological operations are performed on the resulting binary image to obtain a two-dimensional convex hull; at the same time, the lowest ground clearance of the point cloud within each occupied grid is recorded; (c) Each occupied grid is converted into a three-dimensional pseudopoint, that is, the X and Y coordinates of the grid center are used as the horizontal position, and the recorded lowest height is used as the Z value; (d) The three-dimensional pseudopoints are grouped by a three-dimensional clustering algorithm to form independent obstacle clusters; (e) Each three-dimensional cluster is projected onto the horizontal plane and mapped and matched with the two-dimensional convex hull; after successful matching, the lowest Z value of all three-dimensional pseudopoints in the three-dimensional cluster is used as the lowest ground clearance of the three-dimensional obstacle represented by this two-dimensional convex hull.

[0008] Further, the two-dimensional panoramic image in S2 is obtained through the following sub-steps: (2.1) Acquire panoramic images taken by multiple cameras at the same time; (2.2) Perform distortion correction processing on the panoramic images of each camera according to the intrinsic parameters obtained by calibration in advance; then, project each image after distortion correction onto a unified cylindrical surface model through perspective transformation according to the extrinsic parameter matrix of each camera relative to the center of the vehicle to obtain a projection image; finally, align and superimpose multiple projection images on the cylindrical surface.

[0009] Further, step S3 includes the following sub-steps: S3.1: Project the 3D general obstacle detection result of S1 onto the 2D panoramic image and perform association matching with the 2D detection box output by S2. The 3D obstacle that is successfully matched is assigned a semantic category label of the corresponding 2D detection box, and those that fail to match are uniformly marked as "other" category; S3.2: Compare the projection box of the 3D obstacle with the assigned semantic label with the 2D panoramic semantic segmentation map obtained by S2 at the pixel level; traverse all pixels in the projection box and count its semantic label distribution; when the projection box contains multiple connected regions with different semantic labels, the 3D obstacle is determined to be an adhered body; S3.3: Backproject the boundaries of multiple connected regions with different semantic labels in the 2D panoramic semantic segmentation map of the adhered body to the 3D point cloud space to form segmentation constraints; S3.4: Use the segmentation constraints to cluster and segment the point cloud of the adhered 3D obstacle, thereby splitting the projection box of the single 3D obstacle into multiple independent 3D targets corresponding to different semantic entities.

[0010] Furthermore, the method also includes: S4: classifying and optimizing the 3D obstacles after semantic enhancement processing in S3, filtering out false detections and spurious detections, and determining the most reliable category label for each obstacle that is stably tracked.

[0011] Further, S4 specifically includes: S4.1: For dynamic obstacles, Kalman filtering is used for motion state estimation and short-term trajectory prediction; for static obstacles, their position and semantic information are injected into the persistent environment map for stability confirmation and environment map update; S4.2: All obstacles detected in the current frame are aligned with the world coordinate system, and the obstacles in the current frame are matched with the tracked obstacle trajectories in the previous frame using a target association algorithm to form a continuous tracking chain; for successfully matched obstacles, their states are smoothly updated using a filtering algorithm; S4.3: All 3D obstacle detection boxes in the current frame are... The system is registered with a pre-stored high-precision semantic map. If an obstacle is located in a non-physically reasonable area of ​​the map, it is judged as a false detection and removed. If the obstacle is located outside the map, it is directly filtered out. S4.4: Cross-frame association and trajectory modeling are performed for each obstacle. Obstacles that only appear in a single frame or whose motion trajectory does not conform to physical constraints are identified as false detections and filtered out. S4.5: A category confidence queue is maintained for each stably tracked obstacle. By analyzing the classification results of its historical N frames, the Viterbi algorithm is used to determine the most reliable category label at present, so as to eliminate the category flickering phenomenon caused by sensor jitter or recognition model uncertainty.

[0012] A 3D target detection device based on 2D vision enhancement and multi-laser fusion point cloud includes a solid-state LiDAR positioned directly above the front of an autonomous vehicle, multiple single-line mechanical LiDARs positioned around the vehicle body, multiple cameras positioned around the vehicle body, and a target detection system including a general obstacle recognition module, an image stitching module, a target detection and semantic segmentation module, a 2D-3D data fusion and semantic enhancement module, and a hierarchical processing and optimization module. The solid-state LiDAR is used to construct a point cloud of suspended obstacles; the multiple single-line mechanical LiDARs fill in the blind spots of the solid-state LiDAR without blind spots; the multiple cameras collectively cover the 360-degree surrounding environment of the autonomous vehicle; the general obstacle recognition module is used to perform point cloud analysis based on local grid maps and ray tracing methods, respectively analyzing the single-line mechanical LiDARs and the target detection point cloud of the vehicle body. The system uses point clouds scanned by mechanical and solid-state lidar to identify 3D general obstacles and calculates the ground clearance of the 3D general obstacles identified by the solid-state lidar. The image stitching module stitches images from multiple cameras into a 2D panoramic image. The target detection and semantic segmentation module uses a 2D deep learning detection model and a segmentation model to perform target detection and semantic segmentation on the 2D panoramic image, respectively. The 2D-3D data fusion and semantic enhancement module projects the identification results of the 3D general obstacles onto the 2D panoramic image and matches them with the 2D detection boxes output by the target detection and semantic segmentation module. Semantic enhancement processing is performed on the successfully matched 3D general obstacles to obtain independent 3D targets corresponding to different semantic entities.

[0013] Furthermore, the device also includes a hierarchical processing and optimization module, which is used to classify and optimize the semantically enhanced 3D obstacles output by the 2D-3D data fusion and semantic enhancement module, filter out false detections and spurious detections, and determine the most reliable category label for each stably tracked obstacle.

[0014] An electronic device includes: one or more processors; and a storage device for storing one or more programs, which, when executed by the electronic device, enable the electronic device to implement a three-dimensional target detection method based on two-dimensional vision-enhanced multi-laser fusion point clouds.

[0015] The beneficial effects of the present invention are as follows: (1) All-round perception without blind spots: Through the coordinated cooperation of solid-state lidar and single-line lidar, full-range safety protection from the feet to the top of the head is realized, effectively eliminating the perception blind spots.

[0016] (2) Efficient two-dimensional-three-dimensional data fusion: By using panoramic image stitching technology, multiple camera images are combined into one image for deep learning inference, which reduces computational complexity and reduces redundant operations for subsequent matching with three-dimensional obstacles.

[0017] (3) Accurate obstacle height measurement: Through three-dimensional pseudo-point clustering and two-dimensional convex hull mapping matching, the minimum ground clearance of obstacles was accurately measured, providing key data support for the passage decision of unmanned vehicles.

[0018] (4) Strong adaptability to dynamic environment: The single-frame general obstacle detection method can identify both static and dynamic obstacles at the same time. Combined with category-specific tracking and prediction, it significantly improves the perception ability in complex dynamic environments.

[0019] (5) Rich semantic information: The semantics of three-dimensional obstacles are enhanced by the two-dimensional visual detection results, so that the output results not only contain geometric information, but also rich category information, providing a more comprehensive environmental understanding for the downstream decision planning module. Attached Figure Description

[0020] Figure 1 is a flowchart of a three-dimensional target detection method based on two-dimensional vision enhancement and multi-laser fusion point cloud according to an embodiment of the present invention.

[0021] Figure 2 is a flowchart of three-dimensional universal obstacle recognition according to an embodiment of the present invention.

[0022] Figure 3 is a flowchart of panoramic image stitching according to an embodiment of the present invention.

[0023] Figure 4 is a flowchart of two-dimensional to three-dimensional data matching according to an embodiment of the present invention.

[0024] Figure 5 is a schematic diagram of post-processing optimization according to an embodiment of the present invention. Detailed Implementation

[0025] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. The purpose and effects of the present invention will become clearer. It should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0026] The 3D target detection method based on 2D vision enhancement and multi-laser fusion point cloud of the present invention is implemented based on the following hardware configuration: 1. A solid-state lidar is installed directly above the front of the unmanned vehicle. This lidar has non-repeating scanning characteristics and can efficiently construct a detailed point cloud of suspended obstacles. A single-line mechanical lidar is installed directly below the front of the vehicle, on the left side of the rear, and on the right side of the rear. These three lidars are horizontally installed at a low height of the vehicle body, with their scanning planes parallel to the ground, effectively filling in the blind spots of the solid-state lidar and forming a 360-degree "protective circle" without blind spots.

[0027] All lidar units were precisely calibrated to obtain their own intrinsic and extrinsic parameter matrices, laying the foundation for subsequent data fusion.

[0028] 2. The camera hardware is deployed at the front, rear, left side, and center of the right side of the autonomous vehicle, with one camera installed at each of these locations. These four cameras collectively cover a 360-degree surrounding environment. All cameras have undergone precise calibration to obtain their respective intrinsic and extrinsic parameter matrices, laying the foundation for subsequent data fusion.

[0029] As shown in Figure 1, the three-dimensional target detection method based on two-dimensional vision enhancement and multi-laser fusion point cloud of the present invention includes the following steps: Step 1: Perform three-dimensional general obstacle identification on the point cloud scanned by single-line mechanical lidar and solid-state lidar respectively, and calculate the ground height of the three-dimensional general obstacles identified by solid-state lidar.

[0030] As shown in Figure 2, point cloud data from consecutive frames is projected onto a two-dimensional grid map centered on a single-line mechanical lidar (grid resolution can be adjusted according to actual needs). Each grid cell is marked as idle, unknown, or occupied based on statistical point cloud density and occupancy status. Subsequently, drawing inspiration from ray tracing, the propagation path of the laser beam is analyzed by tracing back from the sensor origin to each occupied grid cell. If the beam is terminated within the expected distance, the grid cell is confirmed as a real obstacle. This single-frame universal obstacle detection method can simultaneously identify both static and dynamic obstacles.

[0031] The point cloud scanned by the solid-state LiDAR is first classified into ground points and obstacle points. Then, based on a local grid map and ray tracing method, the obstacle point cloud is identified as a general obstacle, and the ground clearance of the identified general obstacles is calculated. Specifically, this includes: (a) establishing a grid map centered on the solid-state LiDAR, projecting the point cloud scanned by the solid-state LiDAR onto the grid, sorting the points in the same grid by Z-value, and considering the point with the smallest Z-value as a ground point. Then, the ground expression is calculated using a region growing algorithm, and the ground clearance of the remaining points is calculated, thus obtaining the binary classification of ground points and obstacle points.

[0032] (b) Construct a local grid map and use ray tracing technology to identify occupied grids. Binarize the local grid map and perform morphological operations such as erosion and dilation on the resulting binary map to obtain a two-dimensional convex hull. Simultaneously, record the lowest ground clearance of the point cloud within each occupied grid. (c) Convert each occupied grid into a three-dimensional pseudopoint, i.e., use the X / Y coordinates of the grid center as the horizontal position and the recorded lowest height as the Z value. (d) Group these three-dimensional pseudopoints using a three-dimensional clustering algorithm (e.g., Euclidean clustering) to form independent obstacle clusters. (e) Project each three-dimensional cluster onto a horizontal plane and map it to the two-dimensional convex hull obtained through general obstacle detection. After successful matching, use the lowest Z value of all three-dimensional pseudopoints in the three-dimensional cluster as the lowest ground clearance of the three-dimensional obstacle represented by this two-dimensional convex hull.

[0033] Step 2: Stitch together images captured by multiple cameras to form a two-dimensional panoramic image; use a two-dimensional deep learning detection model and segmentation model to perform object detection and semantic segmentation on the two-dimensional panoramic image to obtain two-dimensional detection results.

[0034] Images from four cameras on the autonomous vehicle are stitched together to form a panoramic image. First, distortion correction is applied to each image based on the camera calibration parameters. Then, perspective transformation is used to project the four images onto a uniform cylindrical surface. Finally, stitching seams are blended to create a seamless panoramic image. The panoramic image can be viewed as a wide-angle image taken by a virtual camera, with the intrinsic and extrinsic parameters of the virtual camera obtained based on the calibration information of the four cameras.

[0035] Pre-trained two-dimensional deep learning models (such as YOLOv10, Mask R-CNN, etc.) are used to perform object detection and semantic segmentation on panoramic images, resulting in two-dimensional detection results including bounding boxes, class labels, and segmentation masks.

[0036] As shown in Figure 3, the panoramic image stitching includes the following steps: (2.1) The multi-camera image acquisition and preprocessing system synchronously triggers four cameras to acquire four original panoramic images at the same time.

[0037] (2.2) Panoramic Image Stitching: The four panoramic images are stitched together into a seamless two-dimensional panoramic image. The specific steps are as follows: (2.2.1) Distortion Correction: First, based on the intrinsic parameters (including focal length, principal point coordinates, and distortion coefficients) obtained by each camera through pre-calibration, distortion correction processing is performed on their respective original panoramic images to eliminate radial and tangential distortions caused by lens optical characteristics. This step ensures the geometric accuracy of the image, laying the foundation for subsequent precise stitching.

[0038] (2.2.2) Perspective Transformation and Cylindrical Projection: The four distortion-corrected images are projected onto a unified cylindrical model using perspective transformation, based on the extrinsic parameter matrices of each camera relative to the vehicle center. This step unifies images from different viewpoints into the same continuous coordinate space, effectively reducing perspective distortion during subsequent stitching. The projected image is called the "projected image".

[0039] (2.2.3) Seam Blending: The four projection images are aligned and superimposed on a cylindrical surface. In the overlapping areas of adjacent images, a multi-band fusion algorithm or feathering algorithm is used to eliminate obvious seams caused by exposure differences and color deviations. Finally, a seamless 360-degree 2D panoramic image with natural color transitions and visual consistency is generated.

[0040] (2.3) Two-dimensional environment perception based on deep learning: The panoramic image generated in (2.2) is input into a two-dimensional deep learning model pre-trained on a large dataset to obtain rich two-dimensional environment perception results. The two-dimensional deep learning model can use the efficient YOLOv10 model in combination with a lightweight semantic segmentation branch. The output of the model includes at least the following two types of information: (1) Two-dimensional object detection results: The model outputs the two-dimensional bounding boxes of each salient object in the panoramic image, as well as the category label (such as "pedestrian", "car", "truck") and confidence score corresponding to each bounding box.

[0041] (2) Panoramic semantic segmentation results: The model outputs a segmentation mask with the same resolution as the input panoramic image. This mask assigns a semantic category label (such as "road", "sidewalk", "sky", "vehicle", "pedestrian") to each pixel in the image. This result provides pixel-level environmental semantic information.

[0042] Step 3: Project the 3D general obstacle detection results of S1 onto the 2D panoramic image, and perform association matching with the 2D detection box output in Step 2. Then, perform semantic enhancement processing and optimization on the successfully matched 3D obstacles to obtain independent 3D targets corresponding to different semantic entities.

[0043] As shown in Figure 4, step three includes the following sub-steps: S3.1: Project the three-dimensional general obstacle detection results of S1 onto the two-dimensional panoramic image, and perform association matching with the two-dimensional detection boxes output in step two. The three-dimensional obstacles that are successfully matched are assigned the semantic category label of the corresponding two-dimensional detection box, and those that are not successfully matched are uniformly marked as the "other" category.

[0044] First, coordinate system one and projection are performed. Specifically, based on the sensor parameters obtained through pre-calibration (including the extrinsic parameter matrix between the LiDAR and the camera and the intrinsic parameter matrix of the camera), the point cloud data of the three-dimensional general obstacles in S1 is transformed from the LiDAR coordinate system to the coordinate system of the two-dimensional panoramic image synthesized by multiple cameras. After this step, each three-dimensional obstacle detection box can obtain a corresponding projection area (hereinafter referred to as "three-dimensional projection box") on the two-dimensional panoramic image.

[0045] Then, association matching and semantic assignment based on the intersection-union ratio (IoU) are performed. Specifically, the obtained 3D projected bounding boxes are associated with the 2D detection boxes output in step two. This association matching is preferably calculated based on the IoU. A preset IoU threshold is used (usually set to 0.5, but can be adjusted according to the actual scenario). When the IoU between a 3D projected bounding box and a 2D detection box is greater than this threshold, the match is considered successful. Successfully matched 3D general obstacles are assigned the semantic category label (e.g., "vehicle" or "pedestrian") of the corresponding 2D detection box. Any 3D obstacles that fail to match are uniformly labeled as "other".

[0046] S3.2: To overcome the target adhesion problem caused by sparse point clouds in 3D detection, 2D semantic segmentation results are introduced as the basis for refined processing. Specifically: The projection box of the 3D obstacle with assigned semantic labels is compared pixel-level with the high-resolution 2D panoramic semantic segmentation map obtained in step two; all pixels within the projection box are traversed, and the distribution of their semantic labels is statistically analyzed; when the projection box contains multiple connected regions (i.e., "color blocks") with significantly different semantic labels, the 3D obstacle is determined to be an adhered object, that is, its single 3D detection box actually contains multiple independent physical entities.

[0047] S3.3: Backproject the boundaries of multiple connected regions with different semantic labels in the two-dimensional panoramic semantic segmentation map of the adhesion body to the three-dimensional point cloud space to form segmentation constraints.

[0048] Since the boundaries of two-dimensional pixels themselves do not have depth information, this step usually combines the depth distribution of the original three-dimensional obstacle point cloud to generate an approximate segmentation constraint surface in three-dimensional space.

[0049] S3.4: Cluster the point cloud of the adhered 3D obstacle using segmentation constraints, thereby splitting the projection box of the single 3D obstacle into multiple independent 3D targets corresponding to different semantic entities.

[0050] This step employs a semantically constrained Euclidean clustering algorithm, which considers the semantic attribution derived from 2D segmentation while calculating the distance between points. Ultimately, a point cloud cluster originally belonging to a single 3D detection box is segmented into multiple point cloud subclusters. For each of these subclusters, its 3D bounding box is recalculated, and each new bounding box is assigned a corresponding semantic category label obtained from the 2D semantic segmentation. Thus, a contiguous 3D obstacle is broken down into multiple independent 3D targets, each corresponding to a different semantic entity.

[0051] Step 4: Post-processing optimization and perception result output: The 3D obstacles after semantic enhancement in Step 3 are classified and optimized according to different situations to eliminate false detections and spurious detections, and the most reliable category label is determined for each stably tracked obstacle. As shown in Figure 5, Step 4 includes the following sub-steps: S4.1: Hierarchical tracking: Based on the obtained semantic categories, differentiated processing threads are started. For dynamic category obstacles (such as "pedestrian", "forklift", "handcart"), a Kalman filter-based motion tracker is created and maintained. This tracker uses historical frame data to estimate the obstacle's center position, velocity, acceleration, and other motion states, and predicts its short-term (e.g., the next 1-3 seconds) trajectory. This prediction result is used for subsequent path planning and decision-making. For static category obstacles (such as "shelves", "columns", "walls"), motion prediction is not performed. Instead, position refinement is performed. That is, by fusing multiple frames of observation data, point cloud registration or centroid averaging algorithms are used to optimize the precise position and orientation of their 3D bounding boxes. These are then used as stable environmental elements to update the persistent environmental map for subsequent global positioning and navigation.

[0052] S4.2: Spatiotemporal Alignment and Fusion: Align all obstacles detected in the current frame with the world coordinate system; match the obstacles in the current frame with the trajectories of the obstacles tracked in the previous frame through target association algorithms (such as nearest neighbor association or joint probability data association) to form a continuous tracking chain; for successfully matched obstacles, their state (such as position and velocity) will be smoothly updated through filtering algorithms.

[0053] To address the noise, false detections, and category jitter issues in single-frame detection, the following post-processing workflow is introduced: S4.3: False detection filtering based on prior map: All 3D obstacle detection boxes in the current frame are compared with static category obstacles in the pre-stored prior high-precision semantic map. If an obstacle is significantly located inside a known static structure (such as a wall) or in a non-physically plausible area such as the air, it is judged as a false detection and removed. If the obstacle is located outside the map, it is directly filtered out to reduce the processing requirements of downstream modules.

[0054] Specifically, this involves calculating the volume overlap between obstacles and impassable areas on the map. If most of the volume of an obstacle (e.g., more than 70%) is located inside a wall marked on the map, in the air, or in other physically unreasonable areas, then the obstacle is determined to be a false detection and is removed from the output of this frame.

[0055] S4.4: False Target Filtering Based on Trajectory Consistency: Cross-frame association and trajectory modeling are performed for each obstacle. Targets that only appear in a single frame or whose motion trajectory does not conform to physical constraints (such as instantaneous appearance or sudden velocity change) are identified as false detections and filtered out.

[0056] S4.5: Time-series-based category smoothing: A category confidence queue is maintained for each stably tracked obstacle. Each category confidence queue records the recognition result and confidence of the corresponding stably tracked obstacle for each frame during its lifecycle. By analyzing the classification results of its historical N frames, the Viterbi algorithm is used to determine the most reliable category label to eliminate category flickering caused by sensor jitter or uncertainty in the recognition model.

[0057] For example, if a target is identified as a "pedestrian" in 8 out of the last 10 frames and a "handcart" in 2 frames, after the Viterbi algorithm is used, its final output category will be corrected to "pedestrian," thus effectively avoiding the problem of category label flickering caused by sensor jitter or model uncertainty.

[0058] After the above hierarchical processing and optimization, a list of obstacles is output, containing precise 3D position, physical dimensions, stable semantic category, motion state (for dynamic targets), and unique tracking ID. This result will be sent to the downstream navigation, planning, and control modules for real-time decision-making.

[0059] The following example demonstrates the effectiveness of the method and apparatus of the present invention.

[0060] The following example demonstrates how the method described in this embodiment can effectively detect various obstacles, including low pallets, suspended beams, moving pedestrians, and forklifts, with an accuracy rate exceeding 95% and a false detection rate of less than 3%. In particular, its detection capability for low and suspended obstacles is significantly superior to traditional single-LiDAR solutions, providing a reliable guarantee for the safe operation of intelligent forklifts.

[0061] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A three-dimensional target detection method based on multi-laser fusion point clouds enhanced by two-dimensional vision, characterized in that, include: S1: Based on local grid maps and ray tracing methods, 3D general obstacle recognition is performed on the point clouds scanned by single-line mechanical LiDAR and solid-state LiDAR, respectively, and the ground clearance of the 3D general obstacles recognized by the solid-state LiDAR is calculated; S2: Images captured by multiple cameras are stitched together to form a 2D panoramic image; a 2D deep learning detection model and segmentation model are used to perform target detection and semantic segmentation on the 2D panoramic image to obtain 2D detection results; S3: The 3D general obstacle recognition results of S1 are projected onto the 2D panoramic image and matched with the 2D detection boxes output by S2. The matched 3D general obstacles are semantically enhanced to obtain independent 3D targets corresponding to different semantic entities.

2. The three-dimensional target detection method based on two-dimensional vision enhancement and multi-laser fusion point cloud as described in claim 1, characterized in that, Based on local grid maps and ray tracing methods, the point cloud scanned by a single-line mechanical lidar is used for general obstacle identification. Specifically, the following sub-steps are included: (1) Projecting the point cloud data of consecutive frames onto a two-dimensional grid map centered on the single-line mechanical lidar; (2) Marking each grid cell as idle, unknown, or occupied by statistically analyzing the point cloud density and occupancy status; (3) Based on the ray tracing method, tracing back from the sensor origin to each occupied grid cell, analyzing the propagation path of the laser beam, and if the beam is terminated within the expected distance, the grid cell is confirmed as a real obstacle.

3. The three-dimensional target detection method based on two-dimensional vision enhancement and multi-laser fusion point cloud according to claim 1, characterized in that, Based on local grid maps and ray tracing methods, this paper performs general obstacle recognition and calculates the ground clearance of point clouds scanned by solid-state LiDAR. Specifically, it includes the following sub-steps: (a) Establishing a grid map centered on the solid-state LiDAR, projecting the point cloud scanned by the solid-state LiDAR onto the grid, sorting the points in the same grid by Z-value, considering the point with the smallest Z-value as the ground point, then using a region growing algorithm to calculate the ground expression, and finally calculating the ground clearance of the remaining points, thus obtaining a binary classification of ground points and obstacle points; (b) Constructing a local grid map and using ray tracing technology to identify occupied grids, and then processing the local grid map... Binarization is performed, and then morphological operations are performed on the resulting binary image to obtain a two-dimensional convex hull; at the same time, the lowest ground clearance of the point cloud within each occupied grid is recorded; (c) Each occupied grid is converted into a three-dimensional pseudopoint, that is, the X and Y coordinates of the grid center are used as the horizontal position, and the recorded lowest height is used as the Z value; (d) The three-dimensional pseudopoints are grouped by a three-dimensional clustering algorithm to form independent obstacle clusters; (e) Each three-dimensional cluster is projected onto the horizontal plane and mapped and matched with the two-dimensional convex hull; after successful matching, the lowest Z value of all three-dimensional pseudopoints in the three-dimensional cluster is used as the lowest ground clearance of the three-dimensional obstacle represented by this two-dimensional convex hull.

4. The three-dimensional target detection method based on two-dimensional vision enhancement and multi-laser fusion point cloud according to claim 1, characterized in that, The two-dimensional panoramic image in S2 is obtained through the following sub-steps: (2.1) Acquire panoramic images taken by multiple cameras at the same time; (2.2) Perform distortion correction processing on the panoramic images of each camera according to the intrinsic parameters obtained by calibration in advance; then, project each image after distortion correction onto a unified cylindrical surface model through perspective transformation according to the extrinsic parameter matrix of each camera relative to the center of the vehicle to obtain a projection image; finally, align and superimpose multiple projection images on the cylindrical surface.

5. The three-dimensional target detection method based on two-dimensional vision enhancement and multi-laser fusion point cloud according to claim 1, characterized in that, S3 includes the following sub-steps: S3.1: Project the 3D general obstacle detection result of S1 onto the 2D panoramic image and perform association matching with the 2D detection box output by S2. The 3D obstacle that is successfully matched is assigned a semantic category label of the corresponding 2D detection box, and those that fail to match are uniformly marked as "other" category; S3.2: Compare the projection box of the 3D obstacle with the assigned semantic label with the 2D panoramic semantic segmentation map obtained by S2 at the pixel level; traverse all pixels in the projection box and count its semantic label distribution; when the projection box contains multiple connected regions with different semantic labels, the 3D obstacle is determined to be an adhesive body; S3.3: Backproject the boundaries of multiple connected regions with different semantic labels in the 2D panoramic semantic segmentation map of the adhesive body to the 3D point cloud space to form segmentation constraints; S3.4: Use the segmentation constraints to cluster and segment the point cloud of the adhesive 3D obstacle, thereby splitting the projection box of the single 3D obstacle into multiple independent 3D targets corresponding to different semantic entities.

6. The three-dimensional target detection method based on two-dimensional vision enhancement and multi-laser fusion point cloud according to claim 1, characterized in that, The method also includes: S4: classifying and optimizing the 3D obstacles after semantic enhancement in S3, filtering out false detections and spurious detections, and determining the most reliable category label for each obstacle that is stably tracked.

7. The three-dimensional target detection method based on two-dimensional vision enhancement and multi-laser fusion point cloud according to claim 6, characterized in that, S4 specifically includes: S4.1: For dynamic obstacles, Kalman filtering is used for motion state estimation and short-term trajectory prediction; for static obstacles, their position and semantic information are injected into the persistent environment map for stability confirmation and environment map update; S4.2: All obstacles detected in the current frame are aligned with the world coordinate system, and the obstacles in the current frame are matched with the tracked obstacle trajectories in the previous frame using a target association algorithm to form a continuous tracking chain; for successfully matched obstacles, their states are smoothly updated using a filtering algorithm; S4.3: All 3D obstacle detection boxes in the current frame are compared with pre-stored... The system registers the obstacle with a high-precision semantic map. If the obstacle is located in a non-physically reasonable area of ​​the map, it is judged as a false detection and removed. If the obstacle is located outside the map, it is directly filtered out. S4.4: Cross-frame association and trajectory modeling are performed for each obstacle. Obstacles that only appear in a single frame or whose motion trajectory does not conform to physical constraints are identified as false detections and filtered out. S4.5: A category confidence queue is maintained for each stable tracked obstacle. By analyzing the classification results of its historical N frames, the Viterbi algorithm is used to determine the most reliable category label at present, so as to eliminate the category flickering phenomenon caused by sensor jitter or recognition model uncertainty.

8. A three-dimensional target detection device based on multi-laser fusion point clouds enhanced by two-dimensional vision, characterized in that, The system includes a solid-state LiDAR positioned directly above the front of the autonomous vehicle, multiple single-line mechanical LiDARs positioned around the vehicle body, multiple cameras positioned around the vehicle body, and a target detection system comprising a general obstacle recognition module, an image stitching module, a target detection and semantic segmentation module, a 2D-3D data fusion and semantic enhancement module, and a hierarchical processing and optimization module. The solid-state LiDAR is used to construct a point cloud of suspended obstacles. The multiple single-line mechanical LiDARs fill in the blind spots of the solid-state LiDAR's scanning area without any blind spots. The multiple cameras collectively cover the 360-degree surrounding environment of the autonomous vehicle. The general obstacle recognition module uses local grid maps and ray tracing methods to separate the scanning areas of the single-line mechanical LiDARs and solid-state LiDARs. The system performs 3D general obstacle recognition on the point cloud and calculates the ground clearance of the 3D general obstacles identified by the solid-state LiDAR. The image stitching module stitches images captured by multiple cameras into a 2D panoramic image. The target detection and semantic segmentation module uses a 2D deep learning detection model and a segmentation model to perform target detection and semantic segmentation on the 2D panoramic image, respectively. The 2D-3D data fusion and semantic enhancement module projects the recognition results of the 3D general obstacles onto the 2D panoramic image and performs association matching with the 2D detection boxes output by the target detection and semantic segmentation module. The successfully matched 3D general obstacles are semantically enhanced to obtain independent 3D targets corresponding to different semantic entities.

9. The three-dimensional target detection device based on two-dimensional vision enhancement and multi-laser fusion point cloud according to claim 8, characterized in that, The device also includes a hierarchical processing and optimization module, which is used to classify and optimize the semantically enhanced 3D obstacles output by the 2D-3D data fusion and semantic enhancement module, filter out false detections and false detections, and determine the most reliable category label for each obstacle that is stably tracked.

10. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by the electronic device, cause the electronic device to implement the three-dimensional target detection method based on two-dimensional vision enhancement and multi-laser fusion point cloud as described in any one of claims 1 to 7.

Images