A method for semantic intelligent completion of point cloud data for automotive LiDAR in rainy and foggy weather

Abstract

This invention discloses a method for semantic intelligent completion of automotive LiDAR point clouds in rainy and foggy weather, specifically relating to the field of autonomous driving perception technology. The method first collects temporal point cloud data and inertial navigation information of the vehicle under rainy and foggy conditions to construct a sparse point cloud sequence. Then, based on point cloud density drift, echo energy anomalies, and semantic maps, a sparse-semantic tensor graph is constructed, and contextual structural features are extracted using a temporal graph neural network to generate an affine completion set containing affine transformation parameters. Furthermore, a semantic drift compensation mechanism is used to achieve consistent boundary alignment, reconstructing a pseudo-point cloud completion frame and fusing it with the original point cloud to output a semantically completed enhanced frame. This invention can improve the integrity and semantic accuracy of point clouds under adverse weather conditions, enhance the stability of obstacle detection and path planning, and is suitable for complex environmental perception scenarios in intelligent driving systems.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving perception technology, specifically to a method for semantic intelligent completion of automotive LiDAR point cloud in rainy and foggy weather. Background Technology

[0002] LiDAR (Light Detection and Ranging) is a widely used 3D environmental perception device in autonomous driving systems, capable of providing high-precision spatial distance information. However, under complex weather conditions such as rain and fog, the laser signal is scattered and absorbed by a large number of suspended water droplets and particles during propagation, resulting in significantly sparse laser point cloud data or even complete loss of some parts, which seriously affects the accuracy of the perception system in judging obstacles and road boundaries.

[0003] Especially in heavy rain and fog, the echo points of automotive lidar are almost completely missing on distant and low-reflectivity targets. Furthermore, the raindrops themselves generate a large number of false echoes, which further disrupt the feature extraction of the semantic segmentation model. This can cause the system to misjudge static obstacles as dynamic objects or misidentify lane lines, which can easily lead to risks such as emergency braking or lane departure, seriously threatening driving safety.

[0004] Most existing point cloud completion methods are based on deep learning interpolation or graph neural network reconstruction. While they can restore a certain point density under ideal conditions, they lack effective mechanisms to address point cloud sparseness caused by semantic loss in extreme scenarios such as rain and fog. Especially when vehicles are in remote mountainous areas or tunnel entrances with poor GNSS signals, completion algorithms relying on high-precision maps or multi-sensor fusion strategies often struggle to respond in real time, resulting in perception blind spots. Summary of the Invention

[0005] The purpose of this invention is to provide a method for intelligent semantic completion of point clouds of automotive LiDAR in rainy and foggy weather, so as to overcome the shortcomings of the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for intelligent semantic completion of point clouds of automotive lidar in rainy and foggy weather, comprising:

[0007] S100. Obtain the real-time point cloud frame sequence Praw collected by the LiDAR during the autonomous vehicle's driving in rainy and foggy weather, and obtain the vehicle inertial navigation information It and historical speed sequence Vt corresponding to each frame to construct a temporal point cloud set Pseq.

[0008] S200. Based on the average density threshold and local perturbation gradient of the point cloud frames in the Pseq, calculate the sparse coupling factor Ψt and mark the sparse region Zs.

[0009] S300. For the sparse region Zs, construct the corresponding sparse-semantic tensor graph Gs. Gs is dynamically generated based on the point cloud density drift trend, echo energy anomaly index and prior semantic scene map to form a semantic relationship graph with directional edge weights.

[0010] S400: Based on the constructed sparse-semantic tensor graph Gs, obtain the contextual structure association within the corresponding time period, and output the affine complement Ys containing semantic categories and boundary predictions.

[0011] S500: Project the Ys back to the original point cloud space, perform point-level alignment with the corresponding region in the current frame, construct a semantic drift compensation layer Cs, perform affine harmonic reconstruction and multi-frame consistency reconstruction on the identified semantic boundaries, and generate a pseudo-point cloud completion frame Psynth.

[0012] S600. The Psynth is fused into the lidar point cloud of the current frame to form a semantically completed and enhanced frame (Penhanced).

[0013] Preferably, the calculation of the sparse coupling factor Ψt includes:

[0014] S201. Divide the point cloud of each frame in the temporal point cloud set Pseq into a voxel grid, calculate the point density in each voxel unit, and obtain the full frame density distribution map Dt.

[0015] S202. On Dt, calculate the perturbation gradient map Gt based on the density gradient change between adjacent voxels;

[0016] S203. The density distribution map Dt and the perturbation gradient map Gt are weighted and fused, and a sparse coupling factor Ψt is constructed according to the weighting function.

[0017] S204. On Ψt, the discrimination threshold θΨ is adaptively determined based on the changing trend of the current frame and the historical frames to identify the highly sparse coupling region, which is defined as the sparse region Zs.

[0018] Preferably, the discrimination threshold θΨ is adaptively determined based on the changing trend of the current frame and historical frames on Ψt, including:

[0019] S2041. Extract the sparse coupling factor map of multiple consecutive frames, calculate the time change rate of each pixel, and construct the time difference map ΔΨt.

[0020] S2042. Based on the local extreme value changes of ΔΨt, identify the abnormal regions of the changing gradient, and combine them with the historical distribution of point cloud density to construct a time trend adjustment factor λt.

[0021] S2043. Apply λt to the preset static discrimination threshold θ0 using a function mapping method to generate a dynamic threshold. ;

[0022] S2044. Perform a region segmentation operation on Ψt using θΨ, and identify the continuous region with a sparsity factor greater than the threshold θΨ as the sparse region Zs.

[0023] Preferably, for a sparse region Zs, a corresponding sparse-semantic tensor graph Gs is constructed, including:

[0024] S301. Extract the local point cloud density sequence of each point in the sparse region Zs, and calculate the density drift trend vector of multiple consecutive frames.

[0025] S302. Extract echo energy features from each point in Zs and compare them with the environmental background noise model of the current frame to identify abnormal echo reflection points and construct an energy anomaly label map.

[0026] S303. Combining the preset semantic scene map, perform spatial semantic matching on sparse regions, and assign candidate semantic categories and association weights to each point based on the matching confidence.

[0027] S304. The density drift vector, energy anomaly label and semantic matching result are fused at the tensor level, and a directional and weighted connection relationship is constructed for the edge between points to generate a semantic tensor graph Gs.

[0028] Preferably, spatial semantic matching of sparse regions is performed by combining a preset semantic scene graph, including:

[0029] S3031. Extract a set of candidate semantic templates corresponding to the current GPS coordinates, map fragments and road types from the semantic scene graph;

[0030] S3032. Perform multi-scale spatial matching between the local geometric features in the sparse region Zs and the structural features in the candidate template to generate a preliminary matching score matrix.

[0031] S3033. Calculate the confidence distribution of each point belonging to each semantic category based on the matching score matrix, and determine the main candidate category based on the maximum confidence.

[0032] S3034. Record the matched semantic category and its associated weight for each point, and attach it as a semantic attribute to the node attributes of the sparse-semantic tensor graph Gs.

[0033] Preferably, the output includes the affine complement Ys containing semantic category and boundary prediction, including:

[0034] S401. Input the sparse-semantic tensor graph Gs into the temporal graph neural network model;

[0035] S402. In the model encoding stage, the time alignment mechanism between multi-frame tensor graphs is used to perform time-aware weighted aggregation of the node features of historical frames to form a context structure embedding representation.

[0036] S403. In the decoding stage, the target semantic category and boundary residual offset of each node are predicted according to the context representation, and the affine transformation parameter set is introduced to perform local deformation fitting on the initial point cloud.

[0037] S404. Generate an affine complement Ys containing semantic labels, boundary positions, and affine transformation parameters from the prediction results.

[0038] Preferably, the process of generating an affine complement Ys containing semantic labels, boundary locations, and affine transformation parameters from the prediction results includes:

[0039] S4041. Cluster and group the semantic embedding of each node output by the temporal graph neural network to construct candidate semantic structure regions and identify potential continuous boundary segments or target entity contours.

[0040] S4042. For each candidate region, calculate its principal direction vector, geometric center point, and spatial boundary curvature, and extract them as affine reference features.

[0041] S4043. Combining the boundary residual prediction results of the nodes, the local affine transformation matrix is ​​fitted using the minimum mean square error method. The matrix parameters include rotation angle, scaling factor and translation vector.

[0042] S4044. Combine the affine transformation parameters with the semantic labels and boundary control points of the clustered region to generate a structured affine complement Ys, which is used for point cloud interpolation completion and semantic fusion.

[0043] Preferably, the generation of the pseudo-point cloud complete frame Psynth includes:

[0044] S501. Based on the affine transformation parameters in Ys, the semantic boundary control points are spatially reversed and restored to the original point cloud coordinate system to form a candidate completion point set.

[0045] S502. Perform point-level registration between the candidate completion point set and the original point cloud in the sparse region Zs of the current frame, and use nearest neighbor matching and iterative minimum distance optimization to complete the point-level alignment process.

[0046] S503. Construct a semantic drift compensation layer Cs to perform temporal consistency detection on the semantic boundaries of the same type identified in the current frame and multiple frames before and after, and adjust the distribution of boundary points through a temporal weight fusion strategy.

[0047] S504. Perform interpolation reconstruction on the fused Cs to generate a pseudo point cloud complete frame Psynth.

[0048] Preferably, the construction of the semantic drift compensation layer Cs includes:

[0049] S5031. Extract the set of boundary points belonging to the same semantic category in the current frame and multiple consecutive frames before and after it, and establish a time-related trajectory based on the timestamp and spatial location of the points.

[0050] S5032. Calculate the spatial overlap of the boundary point set of each frame, identify areas with large positional offsets, and determine the drift trend based on the boundary curvature change rate.

[0051] S5033. Based on the time stability and drift amplitude of the boundary points, construct a time weighting function and assign a time decay coefficient to each boundary point to dynamically adjust the spatial position weight of the points.

[0052] S5034. After fusing and weighting the multi-frame boundary point set, interpolation fitting is used to generate a smooth and structurally continuous unified boundary distribution. A semantic drift compensation layer Cs is constructed to guide the reconstruction of pseudo-point clouds.

[0053] Preferably, the Psynth is fused into the lidar point cloud of the current frame to form a semantically complete and enhanced frame (Penhanced), including:

[0054] S601. Synchronize the time and coordinates of the pseudo-point cloud completion frame Psynth and the original point cloud of the current frame.

[0055] S602. Based on semantic labels and point density distribution, points covering sparse regions in Psynth are used as the main completion source, while the remaining regions retain the original point cloud.

[0056] S603. During the fusion process, for points that are spatially adjacent but have inconsistent semantic labels, a confidence-weighted decision is introduced to retain the most credible semantic point.

[0057] S604. After fusion is completed, the semantically enhanced frame "Penhanced" is output, and the semantically enhanced frame "Penhanced" is used for obstacle detection and path planning.

[0058] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0059] 1. This invention constructs a sparse-semantic tensor graph and introduces an affine completion mechanism to achieve intelligent semantic completion of sparse point cloud regions of lidar under adverse weather conditions such as rain and fog. Compared with traditional methods based on interpolation or static completion templates, this invention comprehensively considers point cloud density drift, echo energy anomalies, and prior semantic graph information to generate completion results with directional and structure-aware capabilities, significantly improving the accuracy and stability of recovery for key areas (such as road boundaries and obstacle outlines).

[0060] 2. By introducing a semantic drift compensation layer and a temporal consistency detection mechanism, this invention effectively solves problems such as semantic label jumps, boundary drift, and structural disconnection during point cloud completion, ensuring the consistency of the completed point cloud in both the temporal and spatial domains. The resulting semantically completed enhanced frame not only improves the overall density and semantic resolution of the point cloud but can also be directly integrated into downstream obstacle detection and path planning modules, maintaining stable perception and safety decision-making capabilities under complex weather conditions, thus possessing significant practical deployment value. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0062] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] For examples, please refer to Figure 1 As shown in this embodiment, a method for intelligent semantic completion of automotive LiDAR point clouds in rainy and foggy weather includes:

[0065] S100. Obtain the real-time point cloud frame sequence Praw collected by the LiDAR during the autonomous vehicle's driving in rainy and foggy weather, and obtain the vehicle inertial navigation information It and historical speed sequence Vt corresponding to each frame to construct a temporal point cloud set Pseq.

[0066] S200. Based on the average density threshold and local perturbation gradient of the point cloud frames in the Pseq, calculate the sparse coupling factor Ψt and mark the sparse region Zs.

[0067] S300. For the sparse region Zs, construct the corresponding sparse-semantic tensor graph Gs. Gs is dynamically generated based on the point cloud density drift trend, echo energy anomaly index and prior semantic scene map to form a semantic relationship graph with directional edge weights.

[0068] S400: Based on the constructed sparse-semantic tensor graph Gs, obtain the contextual structure association within the corresponding time period, and output the affine complement Ys containing semantic categories and boundary predictions.

[0069] S500: Project the Ys back to the original point cloud space, perform point-level alignment with the corresponding region in the current frame, construct a semantic drift compensation layer Cs, perform affine harmonic reconstruction and multi-frame consistency reconstruction on the identified semantic boundaries, and generate a pseudo-point cloud completion frame Psynth.

[0070] S600. The Psynth is fused into the lidar point cloud of the current frame to form a semantically completed and enhanced frame (Penhanced).

[0071] In autonomous driving systems, LiDAR (Light Detection and Ranging) serves as a core sensor for 3D perception, providing spatial structure information about the vehicle's surrounding environment. To achieve intelligent completion of point cloud data in rainy and foggy weather, it is first necessary to acquire raw point cloud data collected by the LiDAR under complex weather conditions. Specifically, when a vehicle is driving in rainy or foggy weather, the onboard LiDAR outputs 3D point cloud frames at a fixed frequency, denoted as the real-time point cloud frame sequence Praw. This sequence contains the spatial coordinate information and echo intensity value data of each frame at a fixed timestamp.

[0072] Simultaneously, to perform temporal modeling and spatial correlation of the point cloud sequence, it is necessary to synchronously collect attitude information It provided by the vehicle's inertial navigation unit (IMU), including the vehicle's acceleration, angular velocity, yaw angle, and displacement changes at the corresponding time in each frame. Furthermore, by combining this with the historical velocity sequence Vt recorded by the vehicle control system, the vehicle's motion state and trajectory can be further characterized.

[0073] Finally, the real-time point cloud frame sequence Praw, the corresponding attitude information It, and the velocity data Vt are time-aligned and interpolated to construct a continuous point cloud set Pseq with temporal consistency.

[0074] To achieve intelligent completion of lidar point cloud data in rainy and foggy weather, it is necessary to first identify areas where the sensing performance is significantly degraded under the current environmental conditions. The specific steps are as follows:

[0075] Based on the temporal point cloud set Pseq, density and perturbation analysis are performed on each point cloud frame to calculate the sparse coupling factor Ψt, and the sparse region Zs is adaptively calibrated using this factor.

[0076] The calculation of the sparse coupling factor includes the following steps:

[0077] First, a 3D voxel mesh is created for each frame of the temporal point cloud dataset Pseq. Specifically, each frame's point cloud space is divided into voxel units of fixed size (e.g., 0.2 m × 0.2 m × 0.2 m), and the number of laser points contained within each voxel is counted to obtain the density distribution map Dt for that frame. This map records the point density information in units of voxel indices, reflecting the density of the point cloud in the local space.

[0078] On the density distribution map Dt, the local density gradient, i.e., the perturbation gradient map Gt, is calculated based on the difference in the number of points between adjacent voxels. In this embodiment, the gradient calculation is performed using a three-dimensional Sobel operator or a point density difference operator to obtain the density change rate of each voxel in the x, y, and z directions, thereby quantifying the local perturbation intensity.

[0079] The density distribution map Dt and the perturbation gradient map Gt are weighted and fused to construct a sparse coupling factor Ψt. The fusion employs a linear weighting strategy, whereby for each voxel, the sparse coupling factor is a weighted sum of the density and perturbation values. ; where α is the fusion weighting coefficient, which is usually set between 0.5 and 0.8, and the specific value is dynamically adjusted according to factors such as ambient light and rain / fog concentration.

[0080] After obtaining the sparse coupling factor Ψt, an adaptive threshold strategy is used to divide it into regions, thereby identifying the highly sparse coupling region, defined as the sparse region Zs. The threshold determination process includes the following sub-steps:

[0081] Extract the sparse coupling factor map from the current frame and several consecutive frames preceding it (e.g., the first 3 frames), denoted as . Ψt is used to characterize the dynamic fluctuation trend of the sparse factor at each spatial location.

[0082] The temporal difference map ΔΨt is analyzed to identify local extreme points, i.e., regions where drastic sparsity changes occur. Simultaneously, a temporal trend adjustment factor λt is constructed by combining stability indices (such as variance or maximum fluctuation within a sliding window) for each region in the historical density distribution map of the point cloud. This adjustment factor reflects the degree of sparsity deterioration of the current frame relative to historical frames, and its value is typically normalized to [0.5, 1.5].

[0083] The adjustment factor λt is applied to the preset static discrimination threshold θ0, and the dynamic discrimination threshold θΨ for the current frame is generated through a function mapping relationship. The calculation method is as follows: ; where θ0 is usually an empirical threshold for the sparsity factor, such as 30 (corresponding to a moderate level of density and perturbation), and λt provides dynamic adjustment capability to adapt to the point cloud degradation characteristics under different rain and fog intensities.

[0084] A region segmentation operation is performed on the sparse coupling factor Ψt using a dynamic threshold θΨ. That is, all continuous voxel regions with sparsity factors greater than this threshold are defined as sparse regions Zs.

[0085] To further enhance the context-aware capability of semantic completion in sparse regions, this invention proposes a method for constructing a sparse-semantic tensor graph. This method integrates local density drift, echo energy anomaly features, and prior scene knowledge to form a graph structure with directionality and semantic association, which is then used as input to the subsequent semantic completion network. The specific process is as follows:

[0086] First, the density values ​​of each point in the sparse region Zs within the current frame and several frames before and after it (e.g., ±2 frames) are extracted from the temporal point cloud set to form a density time series. A first-order difference and moving average smoothing algorithm is applied to this series to calculate the density change rate and generate a density drift trend vector, which is used to characterize the dynamic evolution direction of the sparse region.

[0087] For each point in the sparse region Zs, the echo energy (intensity) value of its lidar is extracted and compared with the background noise model of the current frame. The background noise model is constructed using the average echo energy and variance of the non-sparse region. Points whose echo energy deviates from the noise distribution center by more than a certain threshold (e.g., 2 standard deviations) are marked as anomalous reflection points, thereby constructing an energy anomaly label map to indicate whether the point cloud is affected by raindrops or fog droplets.

[0088] By combining the prior semantic scene map of the vehicle's current location, spatial semantic matching is performed on sparse regions. This process includes the following four steps:

[0089] The semantic scene graph extracts a set of candidate semantic templates corresponding to the vehicle's current GPS coordinates, road type (such as urban roads, ramps, tunnel entrances, etc.), and map fragments. The semantic scene graph is a pre-built library of typical road scenes, and each template includes information such as structural topology, boundary morphology, and the distribution of common obstacles.

[0090] Local geometric features are extracted from each point within the sparse region Zs, including but not limited to normal vector direction, edge curvature, and principal axis direction of point cloud distribution. These features are then matched with the structural features in the semantic template at multiple scales. The matching method employs normalized feature similarity calculation, and a matching score matrix is ​​generated through weighted cumulative scores. Each element of the matrix represents the degree of matching between a point and a semantic category of a template.

[0091] Based on the matching score matrix, the softmax activation function is used to generate the confidence distribution of each point for different semantic categories. The semantic category with the highest confidence is selected as the main candidate category for that point, and its corresponding confidence value is recorded as the semantic association weight, with a value range of [0,1].

[0092] The semantic category label and associated weight of each point are attached as attributes to the nodes of the sparse-semantic tensor graph Gs, while preserving the original point's position, density drift and energy anomaly information, thus achieving multi-dimensional semantic embedding.

[0093] Finally, the three dimensions mentioned above—density drift vector, energy anomaly label, and semantic matching result—are fused at the tensor level. The fusion process uses a channel stacking method to construct node feature tensors, and edges are constructed between nodes based on geometric adjacency. Each edge is assigned directionality (e.g., along the density drift direction) and edge weight (e.g., the difference in matching confidence), generating a complete semantic tensor graph Gs with directionality and weights. This graph structure serves as the input to the subsequent semantic completion network.

[0094] To achieve structured completion of sparse point cloud regions in rain and fog, this invention uses a temporal graph neural network model to perform semantic reasoning and boundary reconstruction on a pre-constructed sparse-semantic tensor graph Gs, ultimately generating an affine completion set Ys for interpolation completion. The specific steps are as follows:

[0095] The sparse-semantic tensor graph Gs is input into a pre-trained temporal graph neural network model Modelsg, which is a hybrid structure combining graph attention (GAT) and a temporal encoder, enabling dynamic modeling of spatiotemporal dependencies during node feature aggregation. The model's input consists of node feature tensors (including density shift, energy anomalies, and semantic candidate distributions) and a directional edge-weighted connection matrix.

[0096] During the model's encoding phase, a time alignment mechanism is used to handle the node mapping relationship between the current frame and historical frames. A time-gated weighting function (such as temporal convolution or weighted recurrent unit) is used to aggregate the node features of multiple frames to generate a temporal embedding vector containing historical contextual structure information, which is used to enhance the coherence and structural consistency of semantic prediction.

[0097] During the decoding phase, the neural network uses the aforementioned context embedding vectors to perform semantic classification prediction for each node, outputting its target semantic category. Simultaneously, the model predicts in parallel the boundary residual offset of each node in the original point cloud space, i.e., the direction in which the point should undergo positional correction to conform to the true boundary shape. Furthermore, to recover the spatial structure of local regions, an affine transformation parameter set is predicted for each clustered region, including rotation angles (expressed in radians), scaling factors, and translation vectors, for subsequent point cloud deformation fitting.

[0098] The semantic category, boundary offset, and affine transformation parameters of each node are fused to generate the affine complement Ys. This complement provides a structural reference and initial points for subsequent point cloud interpolation and semantic fusion modules. The construction process of the affine complement Ys includes the following sub-steps:

[0099] The semantic embeddings of nodes output by the graph neural network are subjected to similarity clustering. The DBSCAN or MeanShift clustering algorithm, based on embedding spatial distance, is used to divide adjacent nodes with consistent semantic categories and close geometric locations into candidate semantic structure regions. This operation is used to identify point cloud groups that may constitute continuous boundary segments, obstacle contours, or lane lines.

[0100] For each candidate region, its principal direction vector (obtained through principal component analysis to find the first principal axis), geometric center point, and edge curvature are calculated. Curvature can be obtained by calculating the rate of change of the normal vector within the local neighborhood. These parameters serve as the affine baseline features of the region, used for subsequent deformation fitting and boundary alignment.

[0101] By combining the boundary residual prediction results of each node, a minimum mean square error (LMSE) optimization strategy is adopted to solve the local affine transformation matrix. The transformation matrix includes three parameters: rotation angle (rotation around the principal direction axis), scale factor (adjusting local density), and translation vector (correcting center point offset), which are used to transform the predicted point set into a boundary-aligned state.

[0102] Finally, the affine transformation parameters, along with the category labels and boundary control points in the semantic clustering region, are encapsulated into a structured data format to generate the affine complement Ys.

[0103] To achieve structured completion of the sparse region Zs, after predicting the affine completion whole Ys, it is necessary to map it back to the original point cloud space and fuse multi-frame context information to ensure boundary continuity. This invention generates the pseudo-point cloud completion frame Psynth through the following steps:

[0104] First, the boundary control points and their corresponding affine transformation parameters in the affine completion set Ys are extracted. For each completion point, a spatial inverse mapping is performed using the inverse matrix of the affine transformation to restore it to the original point cloud coordinate system. The inverse affine transformation includes inverting the rotation matrix, taking the reciprocal of the scaling factor, and performing a reverse translation of the translation vector, ultimately obtaining a candidate completion point set that possesses the original spatial distribution of the completion intention.

[0105] The aforementioned candidate complete point set is then registered at the point level with the original point cloud within the sparse region Zs of the current frame. During the registration process, a nearest neighbor search strategy based on spatial Euclidean distance is employed, combined with the Iterative Nearest Point (ICP) algorithm, to minimize the average distance error between the complete points and the original points, thus achieving fine point-to-point alignment. The registration output is an aligned set of candidate complete points, exhibiting high spatial consistency.

[0106] To address the potential positional drift of boundary points over time, a semantic drift compensation layer Cs is further constructed to enhance the consistency of the completed boundaries across consecutive frames. This includes the following steps:

[0107] Extract a set of boundary points identified as belonging to the same semantic category from the current frame and several frames before and after it (e.g., ±2 frames), and sort them by timestamp to construct a time series boundary trajectory. The relationship between points is determined jointly by spatial adjacency and category consistency.

[0108] Spatial overlap analysis was performed on the boundary point sets of each frame, and the degree of boundary offset was measured using point cloud voxel overlap rate or Hausdorff distance. If significant drift was detected in a local boundary segment (e.g., exceeding a set threshold of 0.3 meters), the rate of change of its boundary curvature was further calculated to determine whether the offset was caused by actual structural deformation or perception error.

[0109] A time-weighting function is constructed to calculate the stability index (such as positional variance) of each boundary point over time, and a time decay coefficient is generated based on the stability and drift amplitude. The weighting function adopts an exponential decay form:

[0110] Where δt is the time interval between the current point and the reference frame, and γ is a hyperparameter controlling the decay rate. This function is used to dynamically adjust the influence of boundary points during the fusion process.

[0111] A fusion operation is performed on the weighted multi-frame boundary point set, and B-spline interpolation or moving least squares fitting (MLS) algorithm is used to generate structurally continuous boundary curves. These curves constitute a unified semantic boundary contour and are constructed as a semantic drift compensation layer Cs, which is used to guide the boundary reconstruction and semantic assignment of the subsequent pseudo-point cloud completion region.

[0112] On the compensated Cs, pseudo-point cloud interpolation reconstruction is performed. A boundary control point-based triangulation interpolation algorithm or surface fitting method is used to generate a density-continuous, structurally closed complete point cloud frame, Psynth. The generated point cloud not only fills the sparse holes in the original point cloud but also inherits affine structure, semantic labels, and boundary morphology information.

[0113] To achieve semantic enhancement of the current frame of LiDAR data and further improve the perception stability and path evaluation accuracy of the autonomous driving system under extreme weather conditions such as rain and fog, this invention, after generating the pseudo point cloud completion frame Psynth, uses the following steps to fuse it into the original point cloud to construct the semantic completion enhancement frame Penhanced.

[0114] First, the Psynth data is synchronized in time and its coordinates are unified with the original LiDAR point cloud data of the current frame. Time synchronization is based on the timestamp information of each point, and linear interpolation is used to align the completion points collected at different times. Coordinate unification is achieved by calling the extrinsic parameter matrix in the sensor calibration file to transform the Psynth point set from the model generation coordinate system to the vehicle radar coordinate system, ensuring spatial consistency with the original point cloud.

[0115] Before fusion, the point cloud in Psynth is divided into regions based on the semantic labels and spatial locations of the points, and then matched with sparse regions Zs. Within the sparse regions, points in Psynth are preferentially retained as the primary completion source to supplement missing data in the original point cloud; for non-sparse regions, the original point cloud is left unchanged to avoid redundant computation and erroneous replacement. This strategy can be implemented by constructing a spatial mask function, i.e., fusion is performed within the sparse mask region, while other regions remain unchanged.

[0116] During point-level fusion, a confidence-weighted conflict resolution mechanism is introduced for points that are spatially adjacent but have different semantic labels (e.g., the original point is "curbstone" and the completed point is "obstacle"). Specifically, a semantic confidence score is assigned to each point (which can be obtained from the semantic completion network output or historical label stability estimation). Overlapping point pairs are compared using confidence weighting, and only points with higher confidence scores are retained, while conflicting terms are discarded. Confidence fusion can employ a weighted average or maximum value selection strategy.

[0117] After fusion, the semantically completed and enhanced frame "Penhanced" is output. This point cloud frame contains two types of points: one is the original radar acquisition points, and the other is pseudo-point cloud completion points; each point carries spatial coordinates, echo intensity, timestamp, and semantic category label information. Penhanced has higher point density and a more complete semantic structure, and can be used as the perception input for subsequent obstacle detection and path evaluation modules.

[0118] The semantic completion enhancement frame (Penhanced) generated by this invention, after fusing the original LiDAR point cloud with the pseudo-point cloud completion frame (Psynth), possesses higher point cloud density and a more complete semantic structure. Especially in sparse areas under rainy or foggy weather, it effectively enhances the geometric continuity and semantic integrity of road boundaries, static obstacles, and dynamic targets. The Enhanced frame can serve as a key input to the perception system, directly applied to obstacle detection and path planning modules.

[0119] In obstacle detection, the completed point cloud frames improve the ability to identify low-reflectivity targets (such as pedestrians and non-metallic obstacles) and targets partially obscured by rain and fog, enhancing the integrity of target outlines and classification accuracy. In path planning, Enhanced provides structurally continuous road boundaries and semantic partitioning information, which helps in the stable execution of obstacle avoidance strategies, lane keeping, and dynamic replanning, improving the driving safety of autonomous driving systems in adverse weather conditions.

[0120] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for intelligent semantic completion of point clouds for automotive LiDAR in rainy and foggy weather, characterized in that: include: S100. Obtain the real-time point cloud frame sequence Praw collected by the LiDAR during the autonomous vehicle's driving in rainy and foggy weather, and obtain the vehicle inertial navigation information It and historical speed sequence Vt corresponding to each frame to construct a temporal point cloud set Pseq. S200. Based on the average density threshold and local perturbation gradient of the point cloud frames in the Pseq, calculate the sparse coupling factor Ψt and mark the sparse region Zs. The calculation of the sparse coupling factor Ψt includes: S201. Divide the point cloud of each frame in the temporal point cloud set Pseq into a voxel grid, calculate the point density in each voxel unit, and obtain the full frame density distribution map Dt. S202. On Dt, calculate the perturbation gradient map Gt based on the density gradient change between adjacent voxels; S203. The density distribution map Dt and the perturbation gradient map Gt are weighted and fused, and a sparse coupling factor Ψt is constructed according to the weighting function. S204. On Ψt, the discrimination threshold θΨ is adaptively determined based on the changing trend of the current frame and the historical frames to mark the highly sparse coupling region, which is defined as the sparse region Zs. Specifically, on Ψt, the discrimination threshold θΨ is adaptively determined based on the changing trend of the current frame and historical frames, including: S2041. Extract the sparse coupling factor map of multiple consecutive frames, calculate the time change rate of each pixel, and construct the time difference map ΔΨt. S2042. Based on the local extreme value changes of ΔΨt, identify the abnormal regions of the changing gradient, and combine them with the historical distribution of point cloud density to construct a time trend adjustment factor λt. S2043. Apply λt to the preset static discrimination threshold θ0 using a function mapping method to generate a dynamic threshold. ; S2044. Perform a region segmentation operation on Ψt using θΨ, and mark the continuous region with a sparsity factor greater than the threshold θΨ as the sparse region Zs. S300. For the sparse region Zs, construct the corresponding sparse-semantic tensor graph Gs. Gs is dynamically generated based on the point cloud density drift trend, echo energy anomaly index and prior semantic scene map to form a semantic relationship graph with directional edge weights. For the sparse region Zs, a corresponding sparse-semantic tensor graph Gs is constructed, including: S301. Extract the local point cloud density sequence of each point in the sparse region Zs, and calculate the density drift trend vector of multiple consecutive frames. S302. Extract echo energy features from each point in Zs and compare them with the environmental background noise model of the current frame to identify abnormal echo reflection points and construct an energy anomaly label map. S303. Combining the preset semantic scene map, perform spatial semantic matching on sparse regions, and assign candidate semantic categories and association weights to each point based on the matching confidence. S304. Tensor-level fusion of density drift vector, energy anomaly label and semantic matching result, and construction of directional and weighted connection relationship between points to generate semantic tensor graph Gs; S400: Based on the constructed sparse-semantic tensor graph Gs, obtain the contextual structure association within the corresponding time period, and output the affine complement Ys containing semantic categories and boundary predictions. S500: Project the Ys back to the original point cloud space, perform point-level alignment with the corresponding region in the current frame, construct a semantic drift compensation layer Cs, perform affine harmonic reconstruction and multi-frame consistency reconstruction on the identified semantic boundaries, and generate a pseudo-point cloud completion frame Psynth. S600. The Psynth is fused into the lidar point cloud of the current frame to form a semantically completed and enhanced frame (Penhanced).

2. The method for intelligent semantic completion of automotive lidar point clouds in rainy and foggy weather according to claim 1, characterized in that: This involves combining a pre-defined semantic scene graph to perform spatial semantic matching on sparse regions, including: S3031. Extract a set of candidate semantic templates corresponding to the current GPS coordinates, map fragments and road types from the semantic scene graph; S3032. Perform multi-scale spatial matching between the local geometric features in the sparse region Zs and the structural features in the candidate template to generate a preliminary matching score matrix. S3033. Calculate the confidence distribution of each point belonging to each semantic category based on the matching score matrix, and determine the main candidate category based on the maximum confidence. S3034. Record the matched semantic category and its associated weight for each point, and attach it as a semantic attribute to the node attributes of the sparse-semantic tensor graph Gs.

3. The method for intelligent semantic completion of automotive lidar point clouds in rainy and foggy weather according to claim 1, characterized in that: The output includes the affine complement Ys of semantic category and boundary prediction, including: S401. Input the sparse-semantic tensor graph Gs into the temporal graph neural network model; S402. In the model encoding stage, the time alignment mechanism between multi-frame tensor graphs is used to perform time-aware weighted aggregation of the node features of historical frames to form a context structure embedding representation. S403. In the decoding stage, the target semantic category and boundary residual offset of each node are predicted according to the context representation, and the affine transformation parameter set is introduced to perform local deformation fitting on the initial point cloud. S404. Generate an affine complement Ys containing semantic labels, boundary positions, and affine transformation parameters from the prediction results.

4. The method for intelligent semantic completion of automotive lidar point cloud in rainy and foggy weather according to claim 3, characterized in that: The prediction results will generate an affine complement Ys containing semantic labels, boundary locations, and affine transformation parameters, including: S4041. Cluster and group the semantic embedding of each node output by the temporal graph neural network to construct candidate semantic structure regions and identify potential continuous boundary segments or target entity contours. S4042. For each candidate region, calculate its principal direction vector, geometric center point, and spatial boundary curvature, and extract them as affine reference features. S4043. Combining the boundary residual prediction results of the nodes, the local affine transformation matrix is ​​fitted using the minimum mean square error method. The matrix parameters include rotation angle, scaling factor and translation vector. S4044. Combine the affine transformation parameters with the semantic labels and boundary control points of the clustered region to generate a structured affine complement Ys, which is used for point cloud interpolation completion and semantic fusion.

5. The method for intelligent semantic completion of automotive lidar point clouds in rainy and foggy weather according to claim 4, characterized in that: The generation of the pseudo-point cloud complete frame Psynth includes: S501. Based on the affine transformation parameters in Ys, the semantic boundary control points are spatially reversed and restored to the original point cloud coordinate system to form a candidate completion point set. S502. Perform point-level registration between the candidate completion point set and the original point cloud in the sparse region Zs of the current frame, and use nearest neighbor matching and iterative minimum distance optimization to complete the point-level alignment process. S503. Construct a semantic drift compensation layer Cs to perform temporal consistency detection on the semantic boundaries of the same type identified in the current frame and multiple frames before and after, and adjust the distribution of boundary points through a temporal weight fusion strategy. S504. Perform interpolation reconstruction on the fused Cs to generate a pseudo point cloud complete frame Psynth.

6. The method for intelligent semantic completion of automotive lidar point clouds in rainy and foggy weather according to claim 5, characterized in that: The semantic drift compensation layer Cs is constructed as follows: S5031. Extract the set of boundary points belonging to the same semantic category in the current frame and multiple consecutive frames before and after it, and establish a time-related trajectory based on the timestamp and spatial location of the points. S5032. Calculate the spatial overlap of the boundary point set of each frame, identify areas with large positional offsets, and determine the drift trend based on the boundary curvature change rate. S5033. Based on the time stability and drift amplitude of the boundary points, construct a time weighting function and assign a time decay coefficient to each boundary point to dynamically adjust the spatial position weight of the points. S5034. After fusing and weighting the multi-frame boundary point set, interpolation fitting is used to generate a smooth and structurally continuous unified boundary distribution. A semantic drift compensation layer Cs is constructed to guide the reconstruction of pseudo-point clouds.

7. The method for intelligent semantic completion of automotive lidar point clouds in rainy and foggy weather according to claim 6, characterized in that: The Psynth is fused into the lidar point cloud of the current frame to form a semantically complete and enhanced frame, Penhanced, which includes: S601. Synchronize the time and coordinates of the pseudo-point cloud completion frame Psynth and the original point cloud of the current frame. S602. Based on semantic labels and point density distribution, points covering sparse regions in Psynth are used as the main completion source, while the remaining regions retain the original point cloud. S603. During the fusion process, for points that are spatially adjacent but have inconsistent semantic labels, a confidence-weighted decision is introduced to retain the most credible semantic point. S604. After fusion is completed, the semantically enhanced frame "Penhanced" is output, and the semantically enhanced frame "Penhanced" is used for obstacle detection and path planning.

Images