A method and system for laser radar point cloud noise suppression and blind area completion

Abstract

The application relates to the technical field of laser radar point cloud processing, and particularly discloses a laser radar point cloud noise point suppression and blind area completion method and system, wherein the method comprises the following steps: acquiring a current frame point cloud and an image; constructing a space index for the current frame point cloud, performing space density clustering based on a space geometric distance and a laser echo intensity difference value, preliminarily removing noise points, and obtaining a preliminary filtered point cloud; projecting the preliminary filtered point cloud to the current frame image, calculating a pixel proportion of a real obstacle category in each projection area to confirm and delete noise points, and obtaining a noise point filtered point cloud; acquiring an instantaneous local map, performing difference comparison with the noise point filtered point cloud, and completing the point cloud according to a comparison result; dividing the completed points into a plurality of completed areas according to connectivity, calculating a pixel proportion of a real obstacle category in each completed area to screen the completed points, and obtaining an optimized point cloud. Compared with the prior art, the application can reliably complete a random irregular blind area and accurately filter a high-density noise point cluster.

Description

Technical Field

[0001] This invention relates to the field of lidar point cloud processing technology, and in particular to a method and system for lidar point cloud noise suppression and blind zone completion. Background Technology

[0002] In environmental perception systems of intelligent mobile platforms such as autonomous driving and mobile robots, lidar acquires high-precision 3D point cloud data by emitting laser pulses and measuring echo time. However, under adverse weather conditions such as dust, rain, snow, and fog, the laser beam is affected by scattering, absorption, and signal attenuation from suspended particles or precipitation particles in the air, resulting in a significant reduction in effective measurement distance, a significant decrease in point cloud density, and a severe deterioration in echo signal-to-noise ratio. According to Mie scattering theory, the closer the laser wavelength is to the particle size, the more significant the scattering effect. When there are a large number of dust particles in the air with a diameter approximately on the order of the laser wavelength, strong Mie scattering occurs, causing most of the laser energy to be scattered in other directions and unable to effectively return to the receiver. The above effects manifest in two ways at the point cloud level: first, a large number of randomly distributed false noise points, generated by laser scattering on particles, with random spatial positions and echo intensities; second, random and irregular blind zones with no data return, formed by severe laser energy attenuation preventing it from reaching the target or by excessively weak echo signals, which are fundamentally different from fixed geometric blind zones caused by hardware installation. The two types of problems mentioned above will further lead to false alarm detection and missed detection issues, affecting the environmental perception accuracy and autonomous operation security of the mobile platform.

[0003] To address the aforementioned noise issue, existing technologies primarily rely on spatial density clustering of single-frame point clouds for identification. This involves statistically analyzing the number of points within a neighborhood or performing feature calculations on the clustered point clouds using principal component analysis to distinguish noise from real obstacles. However, these methods depend solely on spatial geometric information and do not utilize laser echo intensity information. In environments with high concentrations of particulate matter, the spatial density of high-density noise clusters formed by a large number of noise points may approach or even exceed that of real obstacles, rendering existing methods ineffective in distinguishing them. Chinese patent CN121660914A discloses a LiDAR point cloud denoising method and system based on multi-feature denoising and restoration. This method employs multi-feature denoising and restoration, combining point cloud intensity and distance features for preprocessing, dynamically adjusting the filtering threshold to remove noise, and restoring mistakenly deleted non-noise points through temporal continuity features. Finally, it uses entropy weighting to fuse features for final evaluation, significantly improving positioning accuracy in complex weather conditions and effectively removing noise while preserving environmental feature point clouds. However, this method uses point-by-point dynamic intensity-distance threshold direct filtering, which may lead to more missed detections when noise is dense. Furthermore, the noise filtering in this method is one-time and irreversible. Although a mechanism for correcting accidental deletions is included, the correction relies on temporal density statistics and lacks semantic information support, making it prone to systematic accidental deletion of low-intensity reflection points of real obstacles. In addition, existing technologies mainly compensate for fixed geometric blind spots caused by hardware installation, lacking effective detection and completion methods for detection blind spots that are randomly generated and irregularly change in position under harsh environments. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for noise suppression and blind zone completion in lidar point clouds, aiming to solve the following technical problems: First, existing noise recognition methods rely solely on the spatial geometric density information of a single frame of point cloud. In environments with high concentrations of particulate matter such as dust, rain, snow, and fog, existing methods struggle to effectively distinguish noise from real obstacles. Second, existing technologies are prone to missed detections in noisy scenes, and the noise filtering process is one-time and irreversible. Related error correction mechanisms rely solely on temporal density statistics and lack semantic information support, posing a systematic risk of error correction for low-intensity reflection points of real obstacles. Third, existing technologies mainly compensate for fixed geometric blind zones caused by hardware installation, lacking effective detection and completion methods for randomly generated and irregularly changing detection blind zones in harsh environments. This invention employs a hierarchical denoising mechanism that combines spatial density clustering with a weighted distance function that integrates spatial geometric distance and laser echo intensity difference, and performs secondary verification by combining image semantic segmentation. It also introduces a dynamic blind zone completion mechanism based on the comparison of instantaneous local map and voxel echo intensity difference to improve the accuracy of point cloud noise suppression and the effectiveness of detection blind zone completion in complex environments.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] This invention provides a method for noise suppression and blind zone completion in lidar point clouds, comprising the following steps:

[0007] Simultaneously acquire and preprocess the current LiDAR point cloud data and camera image data to obtain the current frame point cloud and the current frame image;

[0008] A spatial index is constructed for the current frame point cloud. Spatial density clustering is performed on the current frame point cloud using a weighted distance function that fuses spatial geometric distance and laser echo intensity difference. The spatial point density of each point cloud cluster is calculated. Point cloud clusters with spatial point density below the density threshold are marked as candidate noise points and removed from the current frame point cloud to obtain a preliminary filtered point cloud.

[0009] Based on the coordinate transformation relationship between the lidar and the camera, the preliminary filtered point cloud is projected onto the current frame image. The semantic segmentation model is used to calculate the first pixel proportion of the real obstacle category in the projection area. Based on the first pixel proportion, noise is identified and removed from the preliminary filtered point cloud to obtain the noise filtered point cloud.

[0010] A momentary local map is constructed based on the optimized point cloud of the previous frame after motion compensation. A voxel grid is used to compare the noise-filtered point cloud with the momentary local map. Points in the noise-filtered point cloud that are missing but exist in the momentary local map and whose average echo intensity of the corresponding voxels is higher than the intensity threshold are used as completion points to complete the noise-filtered point cloud, thus obtaining the completed point cloud.

[0011] Project all the completed points in the completed point cloud onto the current frame image. Divide the projection of the completed points into several completed regions according to connectivity. Calculate the second pixel proportion of the real obstacle category in each completed region. Based on the second pixel proportion, determine whether to retain or delete the completed points in each completed region to obtain the optimized point cloud.

[0012] The above method incorporates the difference in laser echo intensity into the clustering distance metric during the initial noise screening stage. This allows dust noise (low and random echo intensity) and real obstacles (stable and high echo intensity) to exhibit distinguishable metric differences in their clustering structures, thereby improving the accuracy of the initial screening. Secondary image semantic verification is introduced for the high-density point cloud clusters retained in the initial screening, addressing the problem that single-frame density estimation cannot distinguish between high-density noise clusters and real obstacles. Utilizing the physical fact that dust occlusion does not have spatial continuity between consecutive frames, an instantaneous local map is constructed through motion compensation, and voxel-level difference comparison is performed to achieve automatic detection and completion of random, irregular blind spots. Finally, reverse image semantic verification is performed on the completion results to eliminate misalignment caused by motion estimation errors.

[0013] Furthermore, the spatial density clustering is performed using the Ordered Point Identification Clustering Structure (OPTICS) algorithm, and the spatial index is constructed using a K-tree (KD-tree). The K-tree accelerates neighborhood search, and cluster boundaries are determined based on significant abrupt changes in reachability distances on the reachability distance curve, extracting each point cloud cluster. The OPTICS algorithm does not require pre-specifying the number of clusters and can discover clusters of arbitrary shapes and densities, making it particularly suitable for application scenarios with irregular spatial distribution and uneven density of noise points under severe weather conditions. The K-tree spatial index reduces the time complexity of k-nearest neighbor search from linear to logarithmic, providing an efficient spatial query foundation for rapid clustering of large-scale point clouds.

[0014] Preferably, in the weighted distance function, the spatial geometric distance component is the ratio of the three-dimensional spatial Euclidean distance between two laser points to the maximum value of the neighborhood search radius, and the spatial echo intensity difference component is the ratio of the absolute value of the echo intensity difference between the two laser points to the quantized value of the maximum echo intensity of the lidar. The two components are multiplied by the spatial distance weighting coefficient and its complementary coefficient, respectively, and then summed to obtain the weighted distance function. By normalizing the spatial distance and echo intensity difference to the same interval before weighted summation, spatial distance dominates in clustering, while echo intensity differences can also effectively distinguish between real obstacles and dust noise from the clustering stage.

[0015] Furthermore, the step of confirming and deleting noise in the preliminary filtered point cloud based on the first pixel proportion includes: when the first pixel proportion is lower than a first proportion threshold and the proportion of background or sky category pixels exceeds the background proportion threshold, the point cloud cluster corresponding to the projection area is confirmed as a false noise and deleted; otherwise, the point cloud cluster corresponding to the projection area is retained. This dual-condition judgment strategy can effectively prevent the accidental deletion of real small targets, while ensuring accurate filtering of false noise, forming a two-level noise filtering defense line combining spatial density initial screening and image semantic verification.

[0016] Specifically, the semantic segmentation model is a pixel-by-pixel inference model based on a lightweight network architecture. The output categories of the semantic segmentation model include ground, buildings, vehicles, pedestrians, cyclists, traffic signs, poles, vegetation, and sky. Among these, ground, buildings, vehicles, pedestrians, cyclists, traffic signs, and poles belong to the category of real obstacles. Employing a lightweight network architecture can reduce inference latency while ensuring semantic segmentation accuracy, meeting the requirements of real-time processing.

[0017] Furthermore, the construction of the instantaneous local map includes: reading the optimized point cloud of the previous frame from the cache queue; performing inter-frame point cloud registration between the optimized point cloud of the previous frame and the point cloud of the current frame, and estimating the vehicle's moving speed and angular velocity using extended Kalman filtering; performing coordinate transformation on the optimized point cloud of the previous frame according to the vehicle's moving speed and angular velocity, transforming it to the vehicle's coordinate system at the current moment, to obtain the instantaneous local map. Motion compensation transforms the clean point cloud optimized from the previous frame to the vehicle's coordinate system at the current moment, ensuring that the instantaneous local map and the point cloud of the current frame are in the same coordinate system, providing a reliable spatial reference for voxel-level difference comparison.

[0018] Preferably, the inter-frame point cloud registration is performed using the Generalized Iterative Closest Point (GICP) algorithm, with voxel downsampling preprocessing performed on both frames of point clouds before registration. The Extended Kalman Filter uses a Constant Turning Rate and Velocity (CTRV) model as the motion model, and the observation noise covariance is adaptively adjusted based on the registration residual at the final convergence of the GICP algorithm. The GICP algorithm constructs a probabilistic distance metric using the covariance matrix of the local plane where the point is located, and its registration accuracy and convergence robustness are superior to traditional iterative closest point methods. The Extended Kalman Filter optimizes velocity estimation through the CTRV motion model, effectively suppressing random fluctuations in single-frame registration and improving the smoothness and stability of velocity estimation.

[0019] Furthermore, the voxel side length of the voxel grid is 0.05 meters; the intensity threshold is 15% of the maximum echo intensity quantization value of the lidar. The voxel side length of 0.05 meters takes into account both the spatial resolution of blind zone detection and the sensitivity to point cloud sparsity, enabling accurate positioning of small-sized blind zones while avoiding the generation of fragmented false blind zones; the intensity threshold is set to 15% of the maximum quantization value, which can effectively distinguish between real obstacles with high laser reflectivity and dust noise with significantly low echo intensity.

[0020] Furthermore, the optimized point cloud is cached using a first-in-first-out (FIFO) queue with a capacity of one frame. For the first frame of data after system startup, spatial density clustering is directly performed on the first frame's point cloud to filter noise, and the filtering result is used as the optimized point cloud for the first frame and stored in the FIFO queue. Using a one-frame FIFO queue to cache historical data results in low memory usage, making it suitable for real-time operation on embedded platforms. Furthermore, using only the previous frame ensures a small spatial deviation between the completed area and the current actual scene, avoiding the drift problem caused by accumulated registration errors in multi-frame map accumulation.

[0021] Further, the step of retaining or deleting completion points in each completion region according to the second pixel proportion includes: retaining all completion points in the completion region when the second pixel proportion of the completion region is not lower than the second proportion threshold; deleting all completion points in the completion region when the pixel proportion of the completion region is lower than the second proportion threshold; the division of the completion region adopts the Euclidean distance clustering method based on image pixel coordinates, and aggregating completion points whose projected pixel coordinate distance is less than the pixel distance threshold into the same connected region. By performing image semantic reverse verification on the completion result, and using the latest image of the current frame to perform post-calibration on the completion data from the previous frame, misaligned completion caused by motion estimation error is effectively eliminated, ensuring that the completion result is consistent with the current actual scene in space and semantics.

[0022] This invention also provides a lidar point cloud noise suppression and blind zone completion system, comprising:

[0023] The data acquisition module is used to simultaneously acquire the current LiDAR point cloud data and camera image data and perform preprocessing to obtain the current frame point cloud and the current frame image;

[0024] The noise screening module is connected to the data acquisition module and is used to construct a spatial index for the current frame point cloud. It performs spatial density clustering on the current frame point cloud using a weighted distance function that fuses spatial geometric distance and laser echo intensity difference, calculates the spatial point density of each point cloud cluster, and marks point cloud clusters with spatial point density below the density threshold as candidate noise points, which are then removed from the current frame point cloud to obtain a preliminary filtered point cloud.

[0025] The semantic verification module is connected to the noise screening module. It is used to project the preliminary filtered point cloud onto the current frame image according to the coordinate transformation relationship between the lidar and the camera, calculate the first pixel ratio of the real obstacle category in the projection area using the semantic segmentation model, and confirm and delete noise in the preliminary filtered point cloud according to the first pixel ratio to obtain the noise filtered point cloud.

[0026] The blind spot completion module is connected to the semantic verification module and is used to obtain an instantaneous local map constructed by motion compensation based on the optimized point cloud of the previous frame. The module uses a voxel grid to compare the noise filtering point cloud with the instantaneous local map. The module uses the point cloud of the region that is missing in the noise filtering point cloud but exists in the instantaneous local map and whose average echo intensity of the corresponding voxel is higher than the intensity threshold as completion points and completes the noise filtering point cloud to obtain the completed point cloud.

[0027] The completion verification module is data-connected to the blind zone completion module. It projects all completion points in the completed point cloud onto the current frame image, divides the projection of the completion points into several completion regions according to connectivity, calculates the second pixel proportion of the real obstacle category in each completion region, and determines whether to retain or delete the completion points in each completion region based on the second pixel proportion to obtain the optimized point cloud.

[0028] Furthermore, the system also includes a local storage unit, which contains a first-in-first-out (FIFO) queue with a capacity of one frame for caching the optimized point cloud. The completion verification module is also used to write the obtained optimized point cloud into the FIFO queue, overwriting the earliest historical data. The FIFO queue enables frame-by-frame rolling caching of the optimized point cloud, ensuring the real-time nature of historical data while controlling storage resource usage.

[0029] Furthermore, the system also includes a calibration parameter storage unit for storing the installation extrinsic parameters of the LiDAR relative to the fuselage coordinate system, the installation extrinsic parameters of the camera relative to the fuselage coordinate system, the camera intrinsic parameter matrix, and distortion coefficients. Both the semantic verification module and the completion verification module are data-connected to the calibration parameter storage unit, performing coordinate transformation and projection from the LiDAR coordinate system to the image pixel coordinate system based on the installation extrinsic parameters and the camera intrinsic parameter matrix. Centralizing the calibration parameters in an independent storage unit facilitates unified loading during system initialization and ensures that the semantic verification module and the completion verification module use consistent coordinate transformation parameters, avoiding projection errors caused by parameter inconsistencies.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] (1) This invention uses a weighted distance function that integrates spatial geometric distance and the difference in laser echo intensity to perform spatial density clustering, rather than relying solely on the single dimension of spatial geometric distance. Since noise is usually generated by laser scattering on suspended particles, its echo intensity differs statistically from that of real obstacles. Real obstacles have stable surface reflectivity and high and consistent echo intensity, while dust noise echo intensity is random and low. By incorporating this difference into the clustering distance metric, even when the spatial point density of noise clusters is close to or even exceeds that of real obstacles in a high-concentration particulate environment, the weighted distance function constructed in this invention can still provide an additional dimension for distinguishing noise clusters from real obstacle clusters, thereby effectively improving the problem of insufficient distinguishing ability of pure spatial geometric density methods in such scenarios.

[0032] (2) This invention employs a two-stage cascaded denoising process: first, spatial density clustering is used to initially screen candidate noise points to obtain a preliminary filtered point cloud; then, based on image semantic segmentation, the pixel proportion of the real obstacle category within the projection area is calculated for secondary confirmation and deletion. The initial screening using spatial density clustering is used to narrow down the range of candidate noise points, while the semantic secondary verification stage introduces semantic information of the real obstacle category. Only when the pixel proportion of the real obstacle category within the projection area is low and the pixel proportion of the background or sky category is high is the point cloud cluster confirmed as noise and deleted. Compared to the point-by-point intensity-distance threshold direct filtering, which may cause more missed detections when noise is dense, the hierarchical processing of this invention helps to balance missed detections and false deletions. At the same time, since the final deletion decision of noise points is supported by semantic information rather than simply relying on temporal density statistics, it can reduce the risk of systematic false deletion of low-intensity real obstacle reflection points and make up for the lack of semantic information support in existing false deletion repair mechanisms.

[0033] (3) To address the problems of existing technologies mainly compensating for fixed geometric blind spots caused by hardware installation and lacking detection and completion of randomly generated and irregularly changing detection blind spots in harsh environments, this invention proposes an instantaneous local map constructed based on the optimized point cloud of the previous frame through motion compensation. A difference comparison is then made between the voxel grid and the noise-filtered point cloud of the current frame. Point clouds in areas missing in the current frame but present in the instantaneous local map, and whose corresponding voxel average echo intensity is higher than the intensity threshold, are used as completion points. This mechanism utilizes verified high-confidence point cloud information from historical frames to dynamically detect and complete irregular detection blind spots randomly formed due to laser energy attenuation or weak echo signals. Compared to compensation methods that only target fixed geometric blind spots, this expands the applicable scenarios for blind spot processing, significantly reduces the risk of missed obstacle detection due to blind spots, and improves the security of autonomous operation of the mobile platform.

[0034] (4) After completing the voxel intensity difference completion, the present invention further projects the completion points onto the current frame image, divides the completion region according to connectivity, and calculates the proportion of the second pixel of the real obstacle category in each completion region, thereby confirming whether to retain or delete the completion points in the corresponding completion region. This secondary verification step can perform semantic-level rationality verification on the completed point cloud, which helps to avoid introducing regions that do not belong to real obstacles into the completed point cloud, thereby achieving blind spot completion while taking into account the quality and reliability of the completed point cloud, and improving the reliability and security of blind spot completion.

[0035] (5) This invention uses a first-in-first-out queue with a capacity of one frame to cache historical optimized point clouds. Only one frame of historical data is needed to complete blind spot detection and completion. It has a small memory footprint and avoids the drift problem caused by the accumulation of registration errors in multi-frame cumulative maps. It is suitable for real-time operation on embedded platforms and has good engineering practicality. Attached Figure Description

[0036] Figure 1 This is a flowchart of the method of the present invention;

[0037] Figure 2 This is a schematic diagram of the electronic device structure of the present invention. Detailed Implementation

[0038] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0039] This embodiment provides a method and system for noise suppression and blind zone completion in lidar point clouds. Addressing the issues of randomly distributed false noise (including high-density clustered noise points) and random irregular blind zones in lidar point clouds under adverse weather conditions, the system organically integrates the three-dimensional geometric information of the lidar point cloud, lidar echo intensity information, inter-frame temporal consistency information, and camera visual semantic information. It constructs a four-level cascaded processing link comprising initial spatial density screening, image semantic verification, temporal consistency blind zone completion, and completion semantic confirmation. This achieves high-precision filtering of point cloud noise and reliable completion of random blind zones.

[0040] Reference Figure 1 In one implementation, the method is performed on a mobile platform equipped with a lidar and a camera, and includes the following steps:

[0041] S1, Get the current time The lidar point cloud data and camera image data are preprocessed to obtain the current frame point cloud. and the current frame image .

[0042] Specifically, the main control unit of the mobile platform sends a hardware synchronization pulse signal, simultaneously triggering the LiDAR and camera to acquire data. Each frame of point cloud and each frame of image carries a timestamp tag synchronized by the main control unit to ensure strict alignment of the timestamps of the two sensor data streams. This allows the current time to be obtained from the quadruped robot dog. LiDAR point cloud data and camera image data .

[0043] In one embodiment, the mobile platform is a general-purpose quadruped mobile robot platform equipped with LiDAR and a camera. This embodiment does not limit the specific platform model, and any mobile platform capable of carrying multiple sensors can be used.

[0044] LiDAR point cloud data It contains the three-dimensional spatial coordinates and echo intensity information of each laser point, specifically represented as a quadruple. ,in These are the coordinates of the laser point in the local coordinate system of the lidar. This is the quantized value of the energy intensity of the laser pulse that returns to the receiver after being reflected from the target.

[0045] In one embodiment, the lidar includes, but is not limited to, mechanically rotating lidar, micro-electro-mechanical system (MEMS) solid-state lidar, and hybrid solid-state lidar. This embodiment does not impose any special restrictions on the type of lidar. The camera includes, but is not limited to, wide-angle camera, fisheye camera, and color camera. This embodiment does not impose any special restrictions on the type of camera.

[0046] The system has completed offline calibration of multiple sensors during the initialization phase. Known parameters include: the installation extrinsic parameters of the lidar relative to the fuselage coordinate system. (Including three installation angles: yaw angle, pitch angle, and roll angle, as well as the coordinates of the installation position) ), Installation parameters of the camera relative to the body coordinate system (Including three installation angles: yaw angle, pitch angle, and roll angle, as well as the coordinates of the installation position) ), camera intrinsic parameter matrix (Including horizontal focal length) Vertical focal length Principal point coordinates ) and distortion coefficient vector (including radial distortion coefficient) and tangential distortion coefficient Using the calibration parameters described above, the lidar point cloud can be projected onto the camera image. The specific transformation relationship will be described in detail in the subsequent steps involving projection.

[0047] After acquiring the synchronization data frame, preprocessing is performed on the raw point cloud of the LiDAR: invalid points with coordinate values ​​containing NaN (Not a Number) or infinity are filtered out, and the point cloud is cropped according to the effective ranging range, removing points that exceed the effective ranging range, to obtain the normalized point cloud of the current frame. The distortion correction process is performed on the camera image data using the calibrated distortion coefficients to obtain the current frame image. This ensures that the projection accuracy of the subsequent laser points onto the image is not affected by image distortion.

[0048] S2, for the current frame point cloud A KD-tree spatial index is constructed, and a clustering algorithm based on a weighted distance function that fuses spatial distance and laser echo intensity differences is used to perform sorting point identification. The spatial point density of each point cloud cluster is calculated, and point cloud clusters with spatial point densities below a density threshold are marked as candidate noise points and removed from the current frame point cloud to obtain a preliminary filtered point cloud. .

[0049] Specifically, a KD-tree (K-Dimensional Tree) spatial index is first constructed for all laser points in the current frame. This index data structure recursively partitions the space into a binary tree along the coordinate axes, which reduces the time complexity of the k-nearest neighbor search for any point from linear to linear. Reduced to logarithm This provides an efficient spatial query foundation for rapid clustering of large-scale point clouds.

[0050] Then, the Ordering Points To Identify the Clustering Structure (OPTICS) algorithm is used to cluster the point cloud of the current frame. The OPTICS algorithm does not require pre-specifying the number of clusters and can discover clusters of arbitrary shapes and densities, making it particularly suitable for application scenarios with irregular spatial distribution and uneven density of noise points under adverse weather conditions. During clustering, both spatial geometric distance and laser echo intensity information are considered simultaneously. For any two laser points... and Its weighted distance is defined as:

[0051] ;

[0052] in, The three-dimensional Euclidean distance between two laser points reflects the spatial proximity of the two points on the horizontal plane. The spatial distance weighting coefficient ensures that spatial distance dominates clustering while echo intensity differences can also effectively differentiate clusters. This is the maximum value of the neighborhood search radius, used to normalize the spatial distance to the [0,1] interval; This is the maximum echo intensity quantization value for the lidar (usually 255), used to normalize the echo intensity difference to the [0,1] interval. The reason for introducing echo intensity weight is that: the surface of a real obstacle usually has a relatively stable reflectivity, and its echo intensity distribution is relatively concentrated; while dust noise is generated by the scattering of suspended particles, and its echo intensity distribution is more random and generally lower. By incorporating echo intensity into the distance metric, points with significant differences in echo intensity can be effectively distinguished, which is beneficial for the accurate separation of noise from real obstacles.

[0053] Clustering is performed according to the standard procedure of the sorting point-based clustering structure identification algorithm:

[0054] Step 1: Initialize all points to an unprocessed state and create an empty ordered list of points to store the results sorted by reachability distance;

[0055] Step 2: Traverse all unprocessed points, and use the current point as the center to calculate the distance in the neighborhood radius based on the weighted distance function. Search for adjacent points within the current node; if the number of adjacent points is not less than the minimum number of neighboring points MinPts, then the current point is the core point, calculate the core distance from it to each adjacent point, and insert the adjacent points and their reachable distances into a priority queue based on a min-heap.

[0056] Step 3: Take the point with the smallest reachable distance from the priority queue, search the neighborhood around this point, update the reachable distance of its neighboring points (take the larger value between the core distance of the point and the weighted distance between the two points), and insert the updated neighboring points into the priority queue; mark the point as processed and add it to the ordered point list;

[0057] Step 4: Repeat the above process until the priority queue is empty. If there are still unprocessed points, return to step 2 to continue processing until all points have been processed, and obtain the complete ordered arrangement of all points and their corresponding reachability distance sequence.

[0058] Step 5: Extract clusters based on the reachability distance curve (with the ordered point list index as the horizontal axis and reachability distance as the vertical axis): In the reachability distance curve, clusters correspond to continuous valley areas with low reachability distances. The locations where the reachability distance undergoes a significant abrupt change (i.e., jumps from a low value to a high value) are used as cluster boundaries. Each cluster is extracted from the valleys of the reachability distance curve to obtain the final multiple point cloud clusters. ,in, The number of clusters.

[0059] After clustering, for each point cloud cluster Calculate its spatial point density Spatial statistical features used to distinguish noise from real obstacles:

[0060] ;

[0061] in, For point cloud clusters Number of laser points included For this point cloud cluster The volume occupied in three-dimensional space is calculated by multiplying the side lengths of the axis-aligned bounding boxes (AABBs) of the point cloud cluster in the X, Y, and Z directions, that is: ,in, , These are the coordinates of the largest and smallest vertices in the space of the axis-aligned bounding box, respectively.

[0062] spatial point density With preset density threshold Comparison: If This indicates that the point cloud cluster is sparsely distributed in space, and is more likely a cluster of false noise points generated by scattering from suspended particles. Therefore, it is preliminarily marked as a candidate noise point. If the object is initially identified as a real obstacle, it will be retained, resulting in a preliminary filtered point cloud. In this embodiment, a preset density threshold is used. The value can be adjusted according to the actual number of lidar lines and scanning resolution.

[0063] S3. Based on the coordinate transformation relationship between the LiDAR and the camera, the point cloud clusters retained in the preliminary filtered point cloud are projected onto the current frame image. The semantic segmentation model is used to calculate the first pixel proportion of the real obstacle category in each projection area. Based on the first pixel proportion, noise is confirmed and deleted in the preliminary filtered point cloud to obtain the noise filtered point cloud.

[0064] Specifically, after spatial density filtering in step S2, low-density dispersed noise points have been initially removed. However, in environments with high concentrations of particulate matter (such as sandstorms, mine dust, and other extreme conditions), the spatial density of high-density noise clusters formed by the aggregation of a large number of noise points in space may exceed the density threshold. Since single-frame spatial density estimation alone cannot effectively distinguish them from real obstacles, this step performs secondary image semantic cross-validation on each point cloud cluster retained after filtering in step S2, forming a two-level noise filtering defense line that combines initial spatial density screening with image semantic verification. The specific implementation process is as follows:

[0065] Since the extrinsic parameters of the LiDAR installation, the extrinsic parameters of the camera installation, and the intrinsic parameters of the camera are known after system calibration, candidate noise points can be projected onto the camera image plane for cross-validation through coordinate transformation. For the three-dimensional coordinates in the LiDAR coordinate system... (Homogeneous coordinate representation), firstly, the coordinate transformation matrix from lidar to camera is obtained. Transform it to the camera coordinate system:

[0066] ;

[0067] in, This is a 4×4 homogeneous transformation matrix from the lidar coordinate system to the camera coordinate system. This represents the pose matrix of the lidar in the fuselage coordinate system. This is the pose matrix of the camera in the body coordinate system. The three-dimensional coordinates in the camera coordinate system are obtained. Then, the pinhole camera model is projected onto the normalized image plane and mapped to the pixel coordinate system:

[0068] ;

[0069] in, This is the camera intrinsic parameter matrix. , These are the focal lengths in the horizontal and vertical directions, respectively (unit: pixels). , Principal point coordinates (unit: pixels) This is the depth value of the point in the camera coordinate system (i.e., the distance along the optical axis of the camera). These are the pixel coordinates of the projected image. If the depth value of the candidate noise point after projection... If a point is located behind the camera, it is outside the camera's field of view, and image verification is skipped; processing is then performed directly based on the spatial density result. For candidate noise points projected onto the effective area of ​​the image, their smallest bounding rectangle region on the image is calculated as the region to be detected for image semantic recognition.

[0070] For each region to be detected, a pre-trained deep learning semantic segmentation model is used for semantic feature recognition. Specifically, a lightweight network-based semantic segmentation model is used to perform pixel-by-pixel inference on the input image. Optional network architectures include, but are not limited to, SegFormer, MaskFormer, and HRNet. The predicted categories output by the semantic segmentation model include common outdoor scene categories such as ground, buildings, vegetation, sky, vehicles, pedestrians, cyclists, traffic signs, and poles. The percentage of the first pixel of each real obstacle category (including categories with physical entities such as ground, buildings, vehicles, pedestrians, cyclists, traffic signs, and poles) within the projection area is statistically analyzed. :

[0071] ;

[0072] in, This represents the number of pixels within the projection area that belong to the category of real obstacles. This represents the total number of pixels within the projection area. A two-condition judgment strategy is employed: if... If the background or sky category pixels account for more than 80%, the point cloud cluster corresponding to the projected area is identified as false noise and deleted from the point cloud; otherwise, the corresponding point cloud is retained without deletion. This dual-condition judgment strategy can effectively prevent the accidental deletion of real small targets while ensuring accurate filtering of false noise. After processing in step S3, the noise-filtered point cloud is obtained. The false noise in the point cloud has been effectively removed and can be used for subsequent time consistency detection and blind spot completion steps.

[0073] S4. Obtain the instantaneous local map constructed by motion compensation based on the optimized point cloud of the previous frame. Use voxel mesh to compare the differences between the noise-filtered point cloud and the instantaneous local map. Use the point cloud of the region that is missing in the noise-filtered point cloud but exists in the instantaneous local map and whose average echo intensity of the corresponding voxel is higher than the intensity threshold as the completion point and complete it to the noise-filtered point cloud to obtain the completed point cloud.

[0074] S41 reads the optimized point cloud of the previous frame from the first-in-first-out queue of the local storage unit for subsequent motion estimation.

[0075] Specifically, this embodiment employs a frame-by-frame pipelined processing method. After each frame is processed, the optimized point cloud, obtained through noise filtering and blind spot completion optimization, is written into a first-in-first-out (FIFO) queue with a capacity of one frame in the local storage unit. Earlier historical data is automatically evicted when writing new frames to control memory usage. When processing the current frame... When reading real-time data, it is read directly from the local storage unit. The laser point cloud data has been optimized at any time. This is used in subsequent motion estimation steps. For the first frame of data after system startup (i.e.... Since there is no historical optimization data from the previous frame, the spatial density clustering noise filtering step S2 is directly performed on the current frame, and the filtering result is written into the first-in-first-out queue as the first frame's optimization output for use in the next frame. From the second frame onwards, the complete processing flow is executed normally.

[0076] S42 performs inter-frame point cloud registration between the optimized point cloud of the previous frame and the point cloud of the current frame, and estimates the vehicle's moving speed and angular velocity by combining extended Kalman filtering.

[0077] Specifically, the Generalized Iterative Closest Point (GICP) algorithm is used to optimize the point cloud of the previous frame. and current frame point cloud Registration is performed, and the optimal pose transformation between the two point clouds is solved using a probabilistic matching framework.

[0078] The reason for choosing GICP instead of the traditional Iterative Closest Point (ICP) method is that in quadruped robot dog applications, planar structures such as the ground and walls are the most common geometric features in the environment. GICP constructs a probabilistic distance metric using the covariance matrix of the local plane where a point is located, which can better handle planar structure scenes. Its registration accuracy and convergence robustness are superior to the point-to-point or point-to-surface distance metrics of traditional ICP. The specific implementation steps are as follows:

[0079] First, voxel downsampling preprocessing is performed on both frames of point clouds. The voxel grid size is set to 0.1 meters. The centroid of the point cloud within each voxel is taken as the representative point to reduce the number of points and improve the efficiency of registration calculation, while maintaining the overall spatial structure of the point cloud. After downsampling... Recorded as Downsampling Recorded as .

[0080] Then, a generalized iterative nearest-point registration framework is constructed: first, the target point cloud is... Constructing a KD-tree (K-Dimensional Tree) spatial index accelerates the subsequent nearest neighbor search process, reducing the time complexity of nearest neighbor search from linear to linear. Reduced to logarithm This significantly improves iteration efficiency.

[0081] for Each point in Using KD-trees in Quickly search for it Calculate the local covariance matrix of the nearest neighbors. Based on this estimate The geometric properties of the local plane. Using the plane covariance to construct a probabilistic distance metric from a point to the plane, the registration objective function of the generalized iterative nearest-point algorithm is:

[0082] ;

[0083] in, For the solution to be found coordinate system to The pose transformation matrix of the coordinate system, including the rotation part. Translation section , For the currently used from coordinate system to The pose transformation matrix of the coordinate system. for The Middle The local covariance matrix of each point for The corresponding nearest neighbor point The local covariance matrix.

[0084] The registration objective function is minimized using the Levenberg-Marquardt (LM) nonlinear optimization algorithm. In each iteration, the Jacobian matrix is ​​calculated and the pose parameters are updated. The nearest neighbor search and pose optimization process is repeated iteratively, with each iteration utilizing a KD-tree to accelerate the nearest neighbor search, until the pose transformation increment is less than the convergence threshold (translation increment less than 0.001 meters, rotation increment less than 0.01 degrees) or the maximum number of iterations (50) is reached. After convergence, the final pose transformation matrix is ​​obtained. .

[0085] Based on the converged pose transformation matrix The rotation matrix is ​​obtained by decomposition. Translation vector Translate vector Divide by the time interval between two adjacent frames Obtain the initial linear velocity estimate ,Right now ; rotate matrix The rotation angle is obtained by converting it to an axis-angle representation using the Rodrigues transformation. Rotation angle divided by time interval Obtain the initial angular velocity estimate ,Right now .

[0086] After obtaining the initial velocity estimate, an extended Kalman filter is constructed to optimize the velocity and angular velocity estimates, thereby suppressing single-frame registration noise and improving the smoothness and stability of the velocity estimate. The system state vector is defined as follows: It includes three components: linear velocity. and angular velocity components A constant turn rate and velocity (CTRV) model is adopted as the motion model. This model assumes that the target's angular velocity is constant over a short period of time, and its linear velocity has a constant magnitude but changes direction with the angular velocity. This model is suitable for describing the motion of a quadruped robot dog in non-omnidirectional motion modes. The state transition equation is then:

[0087] ;

[0088] in, For constant turning speed and nonlinear speed state transition function, The time interval between adjacent frames. For process noise, its covariance matrix Let it be a diagonal matrix, with the diagonal elements corresponding to the process noise variance of each state component. The linear velocity noise variance is taken as... Angular velocity noise variance is taken .

[0089] The observation equation is:

[0090] ;

[0091] Among them, observation This refers to the initial velocity estimate obtained from GICP registration, and the observation matrix. for identity matrix For observation noise, its covariance matrix The registration residuals at the final convergence of GICP registration are adaptively adjusted: the larger the registration residuals, the lower the reliability of the registration results, and the observation noise covariance should be increased accordingly.

[0092] In the prediction step, the posterior state estimate from the previous time step is used. Using constant turning rate and speed model Calculate prior state estimation And calculate the Jacobian matrix of the state transition function. Used for covariance prediction; in the update step, current observations are utilized. Calculate Kalman gain Update the posterior state estimate and posterior covariance matrix Finally, the optimal vehicle speed is obtained from the posterior state estimate. and optimal angular velocity The output is used in step S43.

[0093] S43, perform coordinate transformation on the optimized point cloud of the previous frame based on the vehicle's moving speed and angular velocity, and construct the instantaneous local map at the current moment.

[0094] Specifically, based on the optimal vehicle movement speed output in step S42 and angular velocity By performing motion integration on the pose of the previous frame, the vehicle's position at the current moment can be obtained. Pose transformation matrix relative to the previous frame .get Pose transformation matrix at time step Then, cached Time-optimized point cloud Transformed to using coordinate transformation In the current vehicle coordinate system, an instantaneous local map is constructed. Since this invention only needs to utilize the optimized point cloud from the previous frame for temporal consistency detection, only one frame of historical data needs to be cached; the optimized point cloud from the previous frame... Each point in Calculate its coordinates in the current frame coordinate system using the following formula:

[0095] ;

[0096] in, This is the homogeneous coordinate representation of the point cloud in the previous frame in the vehicle coordinate system of the previous frame (4×1 vector, with the fourth dimension being 1). These are the coordinates in the vehicle's coordinate system in the current frame after the transformation. After performing the above coordinate transformation on all points in the optimized point cloud of the previous frame, the instantaneous local map at the current moment is obtained. The map is composed of a motion-compensated optimized point cloud from the previous frame, and is in the same coordinate system as the point cloud of the current frame. Since dust noise is randomly generated, its spatial location lacks temporal consistency between consecutive frames and will not consistently appear in the same spatial location in consecutive frames; while real obstacles (such as the ground, walls, vehicles, pedestrians, etc.) have spatiotemporal continuity and will stably exist in the same or similar spatial locations across multiple frames. Therefore, the instantaneous local map constructed based on the optimized point cloud of the previous frame can highlight the spatial characteristics of real obstacles, suppress noise, and provide a reliable spatial reference benchmark for subsequent voxel-level difference comparisons.

[0097] S44. A voxel grid is used to compare the noise-filtered point cloud with the instantaneous local map. Point clouds in areas that are missing in the noise-filtered point cloud but exist in the instantaneous local map and whose corresponding voxel average echo intensity is higher than the intensity threshold are filled into the noise-filtered point cloud to obtain the filled point cloud.

[0098] Specifically, after the spatial density filtering in step S2 and the semantic verification filtering in step S3, false noise points in the current frame point cloud have been effectively removed. However, due to the severe attenuation of laser energy in harsh environments, there may be a large number of random and irregular blind spots in the current frame where no data is returned because the laser cannot reach the target or the echo signal is too weak. These blind spots are random and irregular in spatial distribution, which is fundamentally different from the fixed geometric blind spots caused by hardware installation (such as the cone-shaped blind spot below the lidar). However, since the distribution of dust particles in space changes rapidly over time, the occlusion effect on the laser usually does not have continuity between two consecutive frames at the same spatial location. That is, a certain spatial area may be occluded by dust in the current frame, forming a blind spot, but it is very likely that it was not occluded in the previous frame and had valid measurement data. Based on this physical fact, the instantaneous local map after motion compensation can be used to detect and complete the blind spots in the current frame. The specific implementation steps are as follows:

[0099] First, filter the noise point cloud at the current moment. and instantaneous local map A comparison was made. A voxel mesh was used to uniformly divide the 3D space, with a side length of 0.05 meters. This size selection comprehensively considered the spatial resolution of blind zone detection and the sensitivity to point cloud sparsity, enabling accurate localization of small blind zones while avoiding fragmented false blind zones: if the voxels are too large, small blind zones cannot be accurately located; if the voxels are too small, they are too sensitive to point cloud sparsity, easily generating fragmented false blind zones.

[0100] Iterate through all non-empty voxels and mark them into three states based on whether they contain point clouds: Type 1 is observed and occupied, i.e., the voxel contains noise-filtered point clouds. At least one point in the data indicates that the spatial region has been effectively observed in the current frame and contains physical entities (real obstacles), thus requiring no further completion; Type 2 is candidate blind zone completion, i.e., the voxel does not contain noise-filtered point clouds. Any point but containing an instantaneous local map At least one point in the voxel indicates that the spatial region was not observed in the current frame (possibly due to dust obscuring a blind spot) but was effectively measured in the previous frame; Type 3 is idle, meaning the voxel is in the point cloud of the current frame. and instantaneous local map None of them contain any points, so no processing is performed.

[0101] For each voxel marked as a candidate blind spot completion region, extract the echo intensity information of all points within that voxel in the instantaneous local map, and calculate the average echo intensity of that candidate completion region. :

[0102] ;

[0103] in, This represents the number of laser points contained within that voxel in the instantaneous local map. For the first The quantized value of the echo intensity at each point.

[0104] The reason for introducing echo intensity judgment is that the surfaces of real obstacles (such as concrete walls, metal car bodies, asphalt roads, etc.) usually have high laser reflectivity, resulting in high echo intensity; while dust noise is generated by scattering from tiny suspended particles, resulting in significantly lower echo intensity. Therefore, an echo intensity threshold is set. The typical value is 15% of the maximum echo intensity quantization value of the lidar (approximately 38 when the maximum quantization value is 255). This threshold setting can effectively distinguish between real obstacles with high laser reflectivity and dust noise with significantly low echo intensity. The specific judgment rules are as follows:

[0105] If the average echo intensity is higher than the intensity threshold, i.e. This indicates that the area in the instantaneous local map corresponds to a real obstacle with high reflectivity. It is confirmed that the area is a blind spot caused by dust occlusion in the current frame and needs to be filled in. All points in the voxel in the instantaneous local map are filled into the noise-filtered point cloud to achieve blind spot filling.

[0106] If the average echo intensity is not higher than the intensity threshold, i.e. This indicates that the area may be low-echo intensity noise left over from the previous frame, and therefore will not be filled in to avoid incorrectly filling the noise into the current frame.

[0107] After traversing and completing all candidate voxels, the completed point cloud is obtained. The point cloud contains two parts: the laser points effectively observed in the current frame and the completion points from the instantaneous map completion to the blind zone in the current frame. It is important to note that this invention only uses the optimized, clean point cloud from the previous frame for blind zone completion, rather than accumulating multiple historical frames to build a dense global map. This design has several advantages: First, maintaining only one frame of historical data results in low memory usage, making it suitable for real-time operation on embedded platforms; second, in continuous motion, using only the previous frame ensures minimal spatial deviation between the completed area and the current actual scene, avoiding drift issues caused by accumulated registration errors over long periods; third, the randomness of dust occlusion means that occluded areas typically only become invalid within one frame, and the data from the previous frame is sufficient to cover the vast majority of blind zone situations.

[0108] S5. Project the completed points added in step S4 into the current frame image. Divide the projection of the completed points into several completed regions according to connectivity. Calculate the second pixel proportion of the real obstacle category in each completed region. Based on the second pixel proportion, retain or delete the completed points in each completed region to obtain the optimized point cloud.

[0109] After the initial blind spot completion based on instantaneous local map difference detection in step S4, two types of incorrect completion may still occur: First, areas that were originally noise in the instantaneous local map may be incorrectly completed in the current frame. Although step S44 has filtered this using an echo intensity threshold, the echo intensity of some noise points (such as large dust clumps) may occasionally exceed the threshold, posing a risk of incorrect completion. Second, errors in vehicle motion estimation may lead to residual discrepancies in the registration between the instantaneous map and the current frame, causing the projected position of the completed point on the image to mismatch with the actual obstacle position. Therefore, this step uses the latest image semantic information captured by the camera in the current frame to finally verify and confirm the completion result of step S4.

[0110] First, extract all the points added in step S44 for completion, and denote them as the completion point set. ,in, For the first One completion point, This represents the total number of newly added laser points. If... If no blind spot requiring completion is detected in the current frame, skip this verification step; if Then, for each completion point, the same coordinate transformation and projection method as in step S3 is applied to project it onto the current frame image. All projection points of the completion points on the image are clustered according to spatial connectivity. A clustering method based on the Euclidean distance of image pixel coordinates is used to aggregate projection points whose projection pixel coordinate distance is less than a preset pixel distance threshold into the same connected region. After connectivity clustering, the completion projection is divided into several disconnected completion regions. Each connected region corresponds to a spatially independent blind spot completion block, facilitating semantic verification one by one. To complete the number of regions.

[0111] For each completed region Calculate its minimum bounding rectangle on the image, and then calculate the proportion of the second pixel of the true obstacle category in the semantic segmentation model prediction results within this rectangular region. :

[0112] ;

[0113] in, This represents the number of pixels within the rectangular area that belong to the category of real obstacles. This represents the total number of pixels within the rectangular area.

[0114] Based on the proportion of the second pixel Determine whether to retain the completed area:

[0115] like This indicates that the corresponding location of the completed area in the image does indeed contain visual features of real obstacles (such as vehicle outlines, building facades, tree trunk textures, etc.), the completion is correct, and all completed points in the connected area are preserved.

[0116] like This indicates that no valid visual features of real obstacles were detected at the corresponding location of the completed area in the image, indicating a completion error (possibly due to noise from incorrect completion or misalignment caused by motion estimation bias). All completed points within this connected region should be deleted.

[0117] Since the completion data comes from the previous frame, while the camera image is the latest captured image of the current frame, image semantic verification can use the latest visual information to perform post-processing calibration on the completion result, effectively overcoming the misalignment problem caused by inter-frame motion estimation errors. After verification in step S5, the optimized point cloud at the current moment, after the complete processing flow, is finally obtained. In this point cloud, false noise has been effectively suppressed, random irregular blind areas have been reliably filled in, and the filled content has been cross-validated using image semantics. The point cloud will be optimized. The data is output to the downstream sensing module and simultaneously written into a first-in-first-out queue, overwriting the oldest historical data for use in the next frame processing cycle.

[0118] In another implementation, the spatial index can be constructed using an octree instead of a K-dimensional tree. An octree constructs a hierarchical index structure by recursively dividing the three-dimensional space into eight equal parts along the three coordinate axes, thus reducing the time complexity of k-nearest neighbor search from... Reduce to This accelerates the neighborhood search process in the sorting point identification clustering structure algorithm, achieving clustering acceleration functionality equivalent to K-dimensional trees. Octrees offer adaptive spatial partitioning advantages in scenarios with extremely uneven point cloud density distributions, making them suitable for applications with significant differences in 3D point cloud density distributions.

[0119] In another implementation, the cache length of the previous frame's optimized point cloud can be extended to N frames (N≥2). Specifically, the first-in-first-out queue capacity is expanded from one frame to N frames. When constructing the instantaneous local map, the cached N frames of historical optimized point clouds are transformed to the current frame's coordinate system and then merged to form a denser historical reference map, further reducing the blind spot detection rate. The choice of the value of N needs to strike a balance between map density (the larger N is, the denser the map) and registration error accumulation (the larger N is, the greater the drift). This approach is suitable for platforms with slower movement speeds, higher registration accuracy, and ample computing resources.

[0120] It is understood that the spatial density clustering algorithm is not limited to the sorting point identification clustering structure algorithm. Other algorithms capable of discovering clusters of arbitrary shapes, such as Density-Based Spatial Clustering of Applications with Noise (DBSCAN), can also be used, as long as their distance metric incorporates both spatial distance and echo intensity information, enabling preliminary labeling of low-density scattered noise points. The semantic segmentation model is not limited to SegFormer, MaskFormer, or HRNet. Other deep learning models capable of outputting pixel-wise category predictions can also be used, including but not limited to the DeepLab series or fully convolutional networks, as long as they can provide reliable pixel-wise semantic predictions in calculating the pixel proportion of real obstacle categories. The inter-frame point cloud registration algorithm is not limited to the generalized iterative nearest-neighbor algorithm. Normal Distributions Transform (NDT) or other registration algorithms capable of estimating the pose transformation between two frame point clouds can also be used, as long as they can provide initial velocity observations for the extended Kalman filter. The mobile platform is not limited to quadrupedal mobile robots; it can also be wheeled mobile robots, autonomous vehicles, or other mobile intelligent agents equipped with LiDAR and cameras, as long as they can perform offline calibration of multiple sensors and achieve time-synchronous acquisition of LiDAR point clouds and camera images. Where there is no conflict, the technical features of the foregoing embodiments can be combined with each other.

[0121] The lidar point cloud noise filtering and blind spot completion system corresponding to the above methods specifically includes:

[0122] The data acquisition module is used to synchronously acquire the current LiDAR point cloud data and camera image data and perform preprocessing to obtain the current frame point cloud and the current frame image.

[0123] The noise screening module is connected to the data acquisition module and is used to construct a spatial index for the current frame point cloud. It performs spatial density clustering on the current frame point cloud using a weighted distance function that fuses spatial geometric distance and laser echo intensity difference, calculates the spatial point density of each point cloud cluster, and marks point cloud clusters with spatial point density below the density threshold as candidate noise points, which are then removed from the current frame point cloud to obtain a preliminary filtered point cloud.

[0124] The semantic verification module is connected to the noise screening module. It is used to project the preliminary filtered point cloud onto the current frame image according to the coordinate transformation relationship between the lidar and the camera, calculate the first pixel ratio of the real obstacle category in the projection area using the semantic segmentation model, and confirm and delete noise in the preliminary filtered point cloud according to the first pixel ratio to obtain the noise filtered point cloud.

[0125] The blind spot completion module is connected to the semantic verification module and is used to obtain an instantaneous local map constructed by motion compensation based on the optimized point cloud of the previous frame. The module uses a voxel grid to compare the noise filtering point cloud with the instantaneous local map. The module uses the point cloud of the region that is missing in the noise filtering point cloud but exists in the instantaneous local map and whose average echo intensity of the corresponding voxel is higher than the intensity threshold as completion points and completes the noise filtering point cloud to obtain the completed point cloud.

[0126] The completion verification module is data-connected to the blind zone completion module. It projects all completion points in the completed point cloud onto the current frame image, divides the projection of the completion points into several completion regions according to connectivity, calculates the second pixel proportion of the real obstacle category in each completion region, and determines whether to retain or delete the completion points in each completion region based on the second pixel proportion to obtain the optimized point cloud.

[0127] In some implementations, the system further includes a local storage unit with a first-in-first-out (FIFO) queue of one frame capacity for caching optimized point clouds. The completion verification module is also used to write the obtained optimized point cloud into the FIFO queue, overwriting the oldest historical data. The FIFO queue enables frame-by-frame rolling caching of the optimized point cloud, ensuring the real-time nature of historical data while controlling storage resource usage.

[0128] In some embodiments, the system further includes a calibration parameter storage unit for storing the installation extrinsic parameters of the LiDAR relative to the fuselage coordinate system, the installation extrinsic parameters of the camera relative to the fuselage coordinate system, the camera intrinsic parameter matrix, and distortion coefficients. Both the semantic verification module and the completion verification module are data-connected to the calibration parameter storage unit, performing coordinate transformation and projection from the LiDAR coordinate system to the image pixel coordinate system based on the installation extrinsic parameters and the camera intrinsic parameter matrix. Centralizing the calibration parameters in a separate storage unit facilitates unified loading during system initialization and ensures that the semantic verification module and the completion verification module use consistent coordinate transformation parameters, avoiding projection errors caused by parameter inconsistencies.

[0129] Electronic devices can be provided as terminals, servers, or other forms of devices.

[0130] Figure 2 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. (Refer to...) Figure 2 Electronic device 100 can be provided as a server or terminal device. (See reference...) Figure 2The electronic device 100 includes a processing component 101, which further includes one or more processors, and memory resources represented by memory 103 for storing instructions, such as application programs, that can be executed by the processing component 101. The application programs stored in memory 103 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 101 is configured to execute instructions to perform the methods described above.

[0131] Electronic device 100 may further include a power supply component 102 configured to perform power management of electronic device 100, a wired or wireless network interface 104 configured to connect electronic device 100 to a network, and an input / output interface 105. Electronic device 100 can operate on an operating system stored in memory 103, such as a Microsoft Server operating system (Windows Server). TM Apple's graphical user interface-based operating system (Mac OS X) TM ), a multi-user, multi-process computer operating system (Unix) TM Linux is a free and open-source Unix-like operating system. TM ), the open-source Unix-like operating system (FreeBSD) TM (or similar.)

[0132] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 103 including computer program instructions that can be executed by a processing component 101 of an electronic device 100 to perform the above-described method.

[0133] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0134] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0135] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0136] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0137] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0138] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0139] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0140] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0141] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0142] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0143] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0144] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0145] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for noise suppression and blind zone completion in lidar point clouds, characterized in that, Includes the following steps: Simultaneously acquire and preprocess the current LiDAR point cloud data and camera image data to obtain the current frame point cloud and the current frame image; A spatial index is constructed for the current frame point cloud. Spatial density clustering is performed on the current frame point cloud using a weighted distance function that fuses spatial geometric distance and laser echo intensity difference. The spatial point density of each point cloud cluster is calculated. Point cloud clusters with spatial point density lower than the density threshold are marked as candidate noise points and removed from the current frame point cloud to obtain a preliminary filtered point cloud. Based on the coordinate transformation relationship between the lidar and the camera, the preliminary filtered point cloud is projected onto the current frame image. The first pixel proportion of the real obstacle category in the projection area is calculated using a semantic segmentation model. Based on the first pixel proportion, noise is identified and removed from the preliminary filtered point cloud to obtain a noise filtered point cloud. A momentary local map is constructed based on the optimized point cloud of the previous frame after motion compensation. A voxel grid is used to compare the noise-filtered point cloud with the momentary local map. Points in the noise-filtered point cloud that are missing but exist in the momentary local map and whose average echo intensity of the corresponding voxels is higher than the intensity threshold are used as completion points to complete the noise-filtered point cloud, thus obtaining the completed point cloud. Project all the completed points in the completed point cloud onto the current frame image. Divide the projection of the completed points into several completed regions according to connectivity. Calculate the second pixel proportion of the real obstacle category in each completed region. Based on the second pixel proportion, determine whether to retain or delete the completed points in each completed region to obtain the optimized point cloud.

2. The method for suppressing noise and completing blind spots in lidar point clouds according to claim 1, characterized in that, The weighted distance function is a weighted sum of the spatial geometric distance component and the laser echo intensity difference component, respectively, with the spatial distance weighting coefficient and its complementary coefficient. The spatial distance component is the ratio of the three-dimensional Euclidean distance between two laser points to the maximum value of the neighborhood search radius, and the laser echo intensity difference component is the ratio of the absolute value of the difference in echo intensity between two laser points to the quantized value of the maximum echo intensity of the lidar.

3. The method for suppressing noise and completing blind spots in lidar point clouds according to claim 1, characterized in that, The step of confirming and deleting noise in the preliminary filtered point cloud based on the first pixel ratio includes: when the first pixel ratio is lower than the first ratio threshold and the background or sky category pixel ratio exceeds the background ratio threshold, confirming that the point cloud cluster corresponding to the projection area is a false noise and deleting it.

4. The method for noise suppression and blind zone completion of lidar point clouds according to claim 1, characterized in that, The semantic segmentation model is a pixel-wise inference model based on a lightweight network architecture. The output categories of the semantic segmentation model include ground, buildings, vehicles, pedestrians, cyclists, traffic signs, poles, vegetation, and sky. Among them, ground, buildings, vehicles, pedestrians, cyclists, traffic signs, and poles belong to the category of real obstacles.

5. The method for noise suppression and blind zone completion of lidar point clouds according to claim 1, characterized in that, The construction of the instantaneous local map includes: reading the optimized point cloud of the previous frame from the cache queue; performing inter-frame point cloud registration between the optimized point cloud of the previous frame and the point cloud of the current frame, and estimating the vehicle's moving speed and angular velocity using extended Kalman filtering; performing coordinate transformation on the optimized point cloud of the previous frame according to the vehicle's moving speed and angular velocity, transforming it to the vehicle's coordinate system at the current moment, and obtaining the instantaneous local map.

6. The method for noise suppression and blind zone completion of lidar point clouds according to claim 5, characterized in that, The inter-frame point cloud registration adopts the generalized iterative nearest point algorithm; the extended Kalman filter uses a constant turning rate and velocity model as the motion model, and the observation noise covariance is adaptively adjusted according to the registration residual when the generalized iterative nearest point algorithm finally converges.

7. The method for suppressing noise and completing blind spots in lidar point clouds according to claim 1, characterized in that, The optimized point cloud is cached using a first-in-first-out queue with a capacity of one frame. For the first frame of data after system startup, spatial density clustering is directly performed on the first frame point cloud to filter noise, and the filtering result is used as the optimized point cloud of the first frame and stored in the first-in-first-out queue.

8. The method for noise suppression and blind zone completion of lidar point clouds according to claim 1, characterized in that, The step of retaining or deleting padding points in each padding region according to the second pixel proportion includes: when the second pixel proportion of the padding region is not lower than the second proportion threshold, retaining all padding points in the padding region; when the pixel proportion of the padding region is lower than the second proportion threshold, deleting all padding points in the padding region; the division of the padding region adopts the Euclidean distance clustering method based on image pixel coordinates, and aggregating padding points whose projected pixel coordinate distance is less than the pixel distance threshold into the same connected region.

9. The method for suppressing noise and completing blind spots in lidar point clouds according to claim 1, characterized in that, The spatial density clustering uses a sorting point identification clustering structure algorithm.

10. A lidar point cloud noise suppression and blind zone completion system, characterized in that, include: The data acquisition module is used to synchronously acquire the current LiDAR point cloud data and camera image data and perform preprocessing to obtain the current frame point cloud and the current frame image. The noise screening module is connected to the data acquisition module and is used to construct a spatial index for the current frame point cloud. It performs spatial density clustering on the current frame point cloud using a weighted distance function that fuses spatial geometric distance and laser echo intensity difference, calculates the spatial point density of each point cloud cluster, and marks point cloud clusters with spatial point density below the density threshold as candidate noise points, which are then removed from the current frame point cloud to obtain a preliminary filtered point cloud. The semantic verification module is connected to the noise screening module. It is used to project the preliminary filtered point cloud onto the current frame image according to the coordinate transformation relationship between the lidar and the camera, calculate the first pixel ratio of the real obstacle category in the projection area using the semantic segmentation model, and confirm and delete noise in the preliminary filtered point cloud according to the first pixel ratio to obtain the noise filtered point cloud. The blind spot completion module is connected to the semantic verification module and is used to obtain an instantaneous local map constructed by motion compensation based on the optimized point cloud of the previous frame. The module uses a voxel grid to compare the noise filtering point cloud with the instantaneous local map. The module uses the point cloud of the region that is missing in the noise filtering point cloud but exists in the instantaneous local map and whose average echo intensity of the corresponding voxel is higher than the intensity threshold as completion points and completes the noise filtering point cloud to obtain the completed point cloud. The completion verification module is data-connected to the blind zone completion module. It projects all completion points in the completed point cloud onto the current frame image, divides the projection of the completion points into several completion regions according to connectivity, calculates the second pixel proportion of the real obstacle category in each completion region, and determines whether to retain or delete the completion points in each completion region based on the second pixel proportion to obtain the optimized point cloud.

Images