Point cloud real-time semantic segmentation automatic driving roadblock detection method and system

Abstract

The present invention is a point cloud real-time semantic segmentation automatic driving roadblock detection method and system, including the following contents: using laser radar to obtain three-dimensional point cloud data of the scene, and marking corresponding labels for each frame of the collected point cloud data to obtain a data set; the labels include fence objects, bicycles, buses, cars, trailers, trucks, motorcycles, sidewalks, grass, and pedestrians; constructing a point cloud segmentation model, which adopts a deep learning architecture PCformer, including a point cloud embedding stage, a sharing stage, and a segmentation head stage; the deep learning architecture PCformer is implemented in Python and accelerated by C++ and CUDA, and after training with the data set, a trained point cloud segmentation model is obtained, which is used to perform real-time detection and classification of roadblock information in an automatic driving scene. The present invention can significantly reduce the running time of the system and realize real-time detection of roadblocks.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned autonomous driving, and in particular to a point cloud real-time semantic segmentation autonomous driving roadblock detection method and system. Background Art

[0002] Autonomous driving can greatly reduce the time people spend on traffic, while providing travel convenience for people who cannot drive, further increasing people's demand for autonomous driving. At the same time, the path planning and traffic optimization capabilities of autonomous driving vehicles can help alleviate traffic congestion, reduce energy consumption, and improve overall traffic efficiency. Autonomous driving technology refers to the ability of vehicles to drive safely without human driver intervention through various sensors, algorithms, and control systems. However, in actual production and life, autonomous driving technology is still in the exploratory stage. Autonomous driving vehicles need to detect and avoid roadblocks in real time, such as pedestrians, other vehicles, construction obstacles, fallen objects, etc., to ensure safe and smooth driving.

[0003] Roadblock detection is a key step for autonomous driving systems to ensure safe driving of vehicles. Failure to detect roadblocks may cause the vehicle to collide, endangering the safety of passengers and pedestrians. Autonomous vehicles usually travel in open and complex environments, where roadblocks are of various types and uncertain locations. It is a challenge to accurately detect various obstacles and help autonomous driving systems more flexibly cope with the diversity in complex environments. At the same time, by accurately detecting roadblocks, the system can plan a safe driving path in real time, bypass roadblocks or take corresponding avoidance measures to ensure smooth driving.

[0004] Roadblock detection relies on the collaborative work of multiple sensors, including LiDAR, cameras, millimeter-wave radars, and ultrasonic sensors. These sensors can provide real-time environmental data, providing a technical basis for detecting and identifying roadblocks. At present, image detection technology, point cloud detection technology, and multimodal fusion are commonly used in the field of roadblock detection, and point cloud detection technology has obvious advantages in three-dimensional spatial information, distance measurement accuracy, environmental adaptability, and real-time performance. Existing point cloud detection technologies include point-based, voxel-based, pseudo-image, and hybrid representation, and hybrid representation is usually better than other methods. However, while hybrid representation brings benefits to performance, it also introduces disadvantages in processing time. The calculation time is significantly longer and is not suitable for real-world scenarios. The present invention proposes a point cloud real-time semantic segmentation automatic driving roadblock detection method to achieve real-time efficiency and enhance generalization ability, which not only improves the real-time detection of roadblocks in automatic driving scenarios, but also transmits the segmentation results to the warning feedback module of automatic driving to complete tasks such as obstacle avoidance, and provides reliable support for the vehicle's automatic perception and decision-making. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for detecting roadblocks in autonomous driving by real-time semantic segmentation of point clouds. The method cleverly utilizes local feature extraction in the process of embedding point clouds and hybrid projection of three-dimensional point clouds and range images, thereby improving accuracy and efficiency. At the same time, the method of the present invention can significantly reduce the running time of the system, thereby realizing real-time detection of roadblocks.

[0006] In order to achieve the above technical problems, the technical solution of the present invention is:

[0007] In a first aspect, the present invention provides a point cloud real-time semantic segmentation automatic driving roadblock detection method, the detection method comprising the following contents:

[0008] Using LiDAR to obtain three-dimensional point cloud data of the scene, including the three-dimensional coordinates of the point in the Cartesian coordinate system, the LiDAR intensity value and the Euclidean distance between the LiDAR emission point and the target object; and labeling each frame of the collected point cloud data with corresponding labels to obtain a data set; the labels include fence class, bicycle class, bus class, car class, trailer class, truck class, motorcycle class, sidewalk class, grass class, and pedestrian class;

[0009] Build a point cloud segmentation model:

[0010] The point cloud segmentation model adopts a deep learning architecture PCformer, which includes a point cloud embedding stage, a sharing stage, and a segmentation head stage;

[0011] After preprocessing the point cloud data, a point cloud P0∈R is obtained. N1×5 , where N1 is the number of points in the point cloud, and 5 is the dimension of the feature of each point; in the point cloud embedding stage, P0 maps the five features to 128 dimensions through a one-dimensional convolutional layer to obtain a high-dimensional feature point cloud P1; at the same time, a K-dimensional tree is constructed on the point cloud P0, and the K-dimensional tree is used to retrieve the K nearest neighbors of each point and construct the tensor P n , tensor P n Then, the neighboring feature point cloud P is obtained after two two-dimensional convolutions and the rectified linear unit activation function. n ∈R K×N1×5 , then, for the neighboring feature point cloud P n Perform the maximum pooling operation to obtain the output point cloud P2∈R N1×128 , point cloud P2 has the same number of points and feature size as P1; the high-dimensional feature point cloud P1 and point cloud P2 are concatenated to obtain point cloud P3, and then point cloud P3 is remapped to a feature size of 128 through one-dimensional convolution. The point cloud after one-dimensional convolution mapping is represented as P e ∈R N1 ×128 ;

[0012] In the sharing phase, Pe As input, the point cloud P is processed through a series of spatial mixing modules and channel mixing modules. e Processing is performed to obtain a merged point cloud; the number of combinations of spatial mixing modules and channel mixing modules in the shared stage is four, and each spatial mixing module processes data on different planes;

[0013] In the segmentation head stage, including one-dimensional convolution and Softmax layer, the merged point cloud output in the sharing stage undergoes a one-dimensional convolution, and then the Softmax layer is used to infer the final label. The output label is the classification result of each point in the point cloud.

[0014] The deep learning architecture PCformer is implemented in Python and accelerated using C++ and CUDA. After training with the dataset, a trained point cloud segmentation model is obtained, which is used for real-time detection and classification of roadblock information in autonomous driving scenarios.

[0015] Furthermore, four spatial mixing modules map the 3D point cloud into different 2D projections. Each spatial mixing module includes a batch normalization module, a mapping module, a two-dimensional deep convolution DW Conv2D, a ReLU activation function, a two-dimensional point-by-point convolution PWConv2D, a remapping module, and a one-dimensional grouped convolution Gp Conv1D.

[0016] The four mapping modules in the four spatial mixing modules map the input point cloud into the orthogonal projection of plane XY, the orthogonal projection of plane XZ, the orthogonal projection of plane YZ and a two-dimensional distance map in sequence; the remapping module is used to restore the 2D projection mapping to a 3D point cloud;

[0017] The channel mixing module includes a batch normalization module, two one-dimensional depthwise convolutions and one-dimensional point-wise convolutions separated by ReLU activation functions, and a one-dimensional grouped convolution;

[0018] The output of the spatial mixing module is jump-connected to the input as the input of the channel mixing module, and the input and output of the channel mixing module are jump-connected as the input of the next level.

[0019] Furthermore, the process of constructing the K-dimensional tree is: for each point in the point cloud P0, find its K neighbors, then stack the 5-dimensional features of these neighbors to form a neighbor feature set, and concentrate the neighbor feature sets of all points in the tensor P n middle.

[0020] Furthermore, the process of preprocessing the point cloud data is: converting the continuous three-dimensional point cloud data into discrete voxels; cropping the point cloud data to the LiDAR field of view, retaining only the points that the LiDAR can see, and discarding the points outside the LiDAR field of view.

[0021] In a second aspect, the present invention provides a point cloud real-time semantic segmentation automatic driving roadblock detection system, the system comprising:

[0022] Point cloud data acquisition module, used to obtain three-dimensional point cloud data of the scene through LiDAR;

[0023] Point cloud data integration module, used to integrate the three-dimensional coordinates of points in the point cloud data, LiDAR intensity values, and the Euclidean distance between the LiDAR emission point and the target object into a point data format;

[0024] Point cloud segmentation model, which is used to predict scene labels for input point clouds and obtain segmentation maps. Each pixel in the segmentation map has a corresponding label and confidence score.

[0025] The label post-processing module is used to perform conflict resolution and boundary smoothing on the results of the point cloud segmentation model. The specific process is:

[0026] 1) Conflict resolution: traverse the segmentation results of the point cloud segmentation model, check whether there are different parts of the same object, define the different segmentation images obtained by the segmentation results as regions, and the labeled objects identified in the region as parts. There are objects with the same label separated by adjacent regions, and different parts of the same labeled object separated in adjacent regions; set a distance threshold. When the pixel labels in two regions are the same but belong to different parts and the distance is less than the distance threshold, it is considered a conflict; for the detected conflict, set a confidence threshold for each type of label, and average the confidence scores of all pixels in the region as the confidence score of the region, and obtain the confidence score of each region. If the confidence score of the current region is higher than the confidence threshold of the current label, retain and merge the labels of the pixel images segmented from the region. If it is not higher, delete the label in the region;

[0027] At the same time, the contour extraction algorithm is used to identify the boundaries of the object and obtain the optimized image after updating the label;

[0028] 2) Boundary smoothing: For the optimized graph after updating the label, each pixel is regarded as a node in the graph, and the adjacent pixels establish edges to build a graph model; define the data item energy function E data And the smoothing term energy function E smooth ;

[0029] Data item energy function E data : ,

[0030] in is the conflict-optimized true label, is the initial segmentation label, is the indicator function;

[0031] Smooth term energy function E smooth : ,

[0032] Where N2 is the set of adjacent pixel pairs, is a pair of adjacent pixels;

[0033] Then set the learning rate and the maximum number of iterations, and use iterative optimization to minimize the data term energy function E data And the smoothing term energy function E smooth , obtain the final optimized smoothing graph;

[0034] The target recognition and classification module is used to identify different types of objects based on the results of the label post-processing module, assign a category label to each identified object, extract the geometric features of the object to calculate the volume of the object, extract the shape features of the object to calculate the aspect ratio and roundness, aggregate the segmentation results to calculate the geometric center C and bounding box of each object, obtain the vehicle's position coordinates V through the GPS sensor, and determine the object's relative position R (R) according to R = C - V x , R y , R z );

[0035] If R x > 0, the object is in front of the vehicle. If R x < 0, the object is behind the vehicle; if R y > 0, the object is on the left side of the vehicle; if R y <0, the object is on the right side of the vehicle; if R z < 0, the object is located under the vehicle; if R z > 0, the object is above the vehicle;

[0036] The target tracking module continuously monitors the state and movement of objects;

[0037] The early warning feedback module is used to monitor the surrounding obstacles in real time through collision risk assessment and early warning triggering, and quickly issue an alarm when potential danger occurs.

[0038] Furthermore, the process of collision risk assessment is:

[0039] Using geometric information, calculate the Euclidean distance D between the center point of the vehicle and the geometric center of the obstacle. Let the coordinates of the center point of the vehicle be P vehicle (x v ,y v ,z v ), the geometric center coordinates of the obstacle are P obstacle (x o ,y o ,z o);

[0040]

[0041] Let the vehicle speed be V v , in meters per second, the obstacle speed is V o , then the relative speed between the vehicle and the obstacle is V relative for:

[0042]

[0043] Set the slow safety distance threshold D safe, low and fast safety distance threshold D safe,high , according to the relative speed V relative And the Euclidean distance D, determine the collision risk level:

[0044] When D>D safe, high is low risk; when D≤D safe, high And D>D safe, low For medium risk; when D≤D safe, low And V relative >0 means high risk.

[0045] Furthermore, the warning triggers include sound and light prompts, dashboard alarms and vibration feedback:

[0046] When D>D safe, high When , it is a low risk and no warning is triggered;

[0047] When D≤D safe, high And D>D safe, low When the medium risk warning is triggered, the light flashes and the medium risk warning message is displayed on the display screen;

[0048] When D≤D safe, low And V relative When >0, a high-risk warning is triggered. Not only will the light flash, a high-risk warning message will be displayed on the display, but the seat will also vibrate as a prompt.

[0049] Further, according to Calculate the volume of an object,

[0050] Among them, A i is the pixel area of ​​each object part, h i is the height of the i-th part of the object, N3 is the total number of parts into which the object is divided;

[0051] The aspect ratio and roundness are:

[0052] Aspect Ratio = , roundness =

[0053] Where A is the area of ​​the object, p is the perimeter; the width and height refer to the width and height of the smallest circumscribed rectangle containing the object, respectively, with the long side of the smallest circumscribed rectangle as the width and the short side as the height;

[0054] Compute the geometric center of all points of each object:

[0055]

[0056] Where C is the geometric center coordinate (C x ,C y ,C z ); N4 is the number of points in the point cloud of an object; P i =(x i ,y i ,z i ) is the coordinate of the i-th point;

[0057] Bounding box: Determine the boundary of the object through the minimum / maximum coordinates, and obtain the width, height, and depth of the bounding box;

[0058] Minimum coordinates:

[0059] Maximum coordinates:

[0060] Bounding box dimensions: .

[0061] Furthermore, the contour extraction algorithm adopts the QuickHull convex hull algorithm, uses the QuickHull convex hull algorithm for processing, obtains multiple convex hull point sets, splices the convex hull point sets into a global convex hull, and once again uses the QuickHull convex hull algorithm to merge the convex hulls.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] The method of the present invention is specifically used for roadblock detection in autonomous driving. It is optimized for point cloud semantic segmentation tasks, cleverly utilizes the extraction of local features in the point cloud embedding process, and combines the hybrid projection technology of three-dimensional point cloud and range image to significantly improve the accuracy and efficiency of detection. In the complex environment of autonomous driving, fast and accurate identification of roadblocks is crucial for driving safety. Therefore, optimized performance and response speed are the core of the present invention.

[0064] In the present invention, by changing the number of network layers of the spatial mixing module and the channel mixing module, the running time of the system can be significantly reduced, so that when the model processes point cloud data, it not only maintains a high segmentation accuracy, but also realizes real-time detection of roadblocks and can quickly respond to changes in the dynamic environment, so that the autonomous driving system can respond in time when facing sudden roadblocks to ensure driving safety.

[0065] In addition, the system of the present invention uses its deep understanding of spatial information to reasonably allocate computing resources during the detection process, thereby further improving the real-time performance of the overall system, providing strong support for the application of autonomous driving technology, and not only enhancing the ability to adapt to complex environments, but also laying the foundation for future intelligent transportation systems. Through the accurate detection and identification of roadblocks, the safety and reliability of the autonomous driving system can be effectively improved, bringing more possibilities for intelligent travel. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a schematic diagram of the overall architecture of a point cloud segmentation model in an embodiment of the present invention;

[0067] Figure 2 It is a schematic diagram of the combined structure of a spatial mixing module and a channel mixing module in one embodiment of the present invention;

[0068] Figure 3 It is a flow chart of the point cloud real-time semantic segmentation automatic driving roadblock detection system of the present invention. DETAILED DESCRIPTION

[0069] The present invention is further explained below in conjunction with the embodiments and drawings, but this is not intended to limit the scope of protection of the present application.

[0070] Embodiment 1:

[0071] The point cloud real-time semantic segmentation automatic driving roadblock detection method of this embodiment includes the following steps:

[0072] 1. Point cloud data collection

[0073] Use LiDAR to obtain 3D point cloud data of the scene. Mainly collect the following ten types of scene data (fence, bicycle, bus, car, trailer, truck, motorcycle, sidewalk, grass, pedestrian), and manually annotate each frame of the collected point cloud data with a specific label.

[0074] 2. Point Cloud Data Integration

[0075] Contains the location coordinates of the point (such as x, y, z) and the LiDAR intensity value and the LiDAR distance (i.e. the Euclidean distance between the LiDAR emission point and the target object).

[0076] 1. Point coordinate acquisition: The polar coordinate data of the LiDAR-generated point includes the Euclidean distance (referred to as distance), azimuth and pitch angle between the LiDAR launch point and the target object. These data are used to convert the polar coordinates of the point into Cartesian coordinates according to the following formula: , , coordinate:

[0077]

[0078] Where r is the Euclidean distance between the LiDAR emission point and the target object, is the horizontal angle (azimuth), is the vertical angle (pitch angle).

[0079] 2. Intensity value and distance acquisition: LiDAR intensity value is directly obtained by LiDAR, distance = , c is the speed of light, and t represents the time required for LiDAR to measure the laser pulse from emission to reception.

[0080] 3. Data integration: The calculated , , The coordinates, distance and intensity values ​​are integrated into a point data format and stored in PCD format.

[0081] 3. Divide the Dataset

[0082] 30,000 frames of point cloud data were obtained using LiDAR, covering the above ten scenes, and the point cloud data was used to construct the dataset R. The dataset R was divided into training set, validation set and test set in a ratio of 7:2:1 for training and validation of the point cloud segmentation model.

[0083] 4. Point cloud data preprocessing is carried out according to the following steps

[0084] 1. Voxelization: Convert continuous 3D point cloud data into discrete voxels (3D pixels), that is, divide the point cloud into small cubic areas to simplify data processing.

[0085] 2. Cropping: Cropping the point cloud data to the sensor (lidar) field of view, reducing sensor outliers and ignoring points that are too far away. In order to retain only the points that the sensor can "see", discarding points outside the sensor's field of view or very far away points can reduce the impact of noise and redundant data and improve the efficiency and accuracy of subsequent processing.

[0086] 5. Build a point cloud segmentation model for feature extraction and segmentation

[0087] The point cloud data is input into the point cloud segmentation model for real-time detection and classification of road obstacle information in autonomous driving scenarios.

[0088] The point cloud segmentation model adopts the deep learning architecture PCformer, and its overall structure is as follows Figure 1 As shown, it includes the point cloud embedding stage, the sharing stage and the segmentation head stage.

[0089] After preprocessing the point cloud data, a point cloud P0∈ is obtained. N1×5 , where N1 is the number of points in the point cloud, and 5 is the dimension of the feature of each point, that is, coordinate, coordinate, Coordinates, LiDAR intensity values, and the Euclidean distance between the LiDAR emission point and the target object; in the point cloud embedding stage, P0 maps the five features to 128 dimensions through a one-dimensional convolution layer, that is, after passing through the one-dimensional convolution layer, the number of features (dimensions) will increase from 5 to 128, thereby obtaining a high-dimensional feature point cloud P1. Then, a K-dimensional tree is constructed on the point cloud P0, and the K-dimensional tree is used to efficiently retrieve the K nearest neighbors of each point and construct local features. Local feature extraction captures information such as shape features, local geometry, and point distribution patterns, enabling the network to learn representations that are robust to scale, direction, arrangement, and density changes.

[0090] Construct a K-dimensional tree (KD tree): For a point in the point cloud P0, find its K neighbors, then stack the 5-dimensional features of these neighbors to construct a neighbor feature set, and concentrate the neighbor feature sets of all points in a tensor P n In this way, the subsequent network processing of these neighbor information becomes more efficient, and the subsequent network processing is used to obtain the neighboring feature point cloud P n ∈R K×N1×5 .

[0091] like Figure 1 As shown, the neighboring feature point cloud P n It is obtained by two 2D convolutions and the rectified linear unit (ReLU) activation function. After that, the neighboring feature point cloud P n Perform a maximum pooling operation, select only the maximum response, and represent the output after the maximum pooling operation as a point cloud P2∈R N1 × 128 , point cloud P2 has the same number of points and feature size as P1.

[0092] Finally, the high-dimensional feature point cloud P1 and point cloud P2 are concatenated to obtain point cloud P3. Then point cloud P3 is remapped to a feature size of 128 through one-dimensional convolution. The point cloud after one-dimensional convolution mapping is represented as P e ∈R N1×128 Then, we enter the feature sharing stage, and use a series of spatial mixing modules and channel mixing modules to analyze the point cloud P.e For processing, the number of combinations of spatial mixing modules and channel mixing modules in the shared stage is four, and each spatial mixing module processes data on different planes.

[0093] like Figure 2 As shown in the figure, in the spatial mixing module, the 3D point cloud is mapped to different 2D projections, including a batch normalization module (Batch Norm), a mapping module (Optimized Projection to plane, four types of mapping modules map the input point cloud to the orthogonal projection of plane XY, the orthogonal projection of XZ and the orthogonal projection of YZ, and a two-dimensional distance map (Range Image)), a remapping module (Re-Projection to point cloud, restores the 2D projection mapping to a 3D point cloud), two two-dimensional depth convolutions (DW Conv2D) separated by ReLU activation functions and two-dimensional point-by-point convolutions (PW Conv2D), and a one-dimensional grouped convolution (Gp Conv1D).

[0094] The structure of the channel mixing module is similar to that of the spatial mixing module, except that the two-dimensional point-by-point convolution and the two-dimensional depth convolution are replaced by one-dimensional point-by-point convolution and one-dimensional depth convolution, and the mapping module and the remapping module are removed. The channel mixing module includes a batch normalization module, two one-dimensional depth convolutions and one-dimensional point-by-point convolutions separated by the ReLU activation function, and a one-dimensional grouped convolution.

[0095] The output of the spatial mixing module is jump-connected to the input as the input of the channel mixing module, and the input and output of the channel mixing module are jump-connected as the input of the next level.

[0096] A series of spatial mixing modules and channel mixing modules are used to analyze the point cloud P e Processing can provide multiple projection perspectives and depth information for point cloud processing, making it easier for the model to extract spatial features more comprehensively. 3D point cloud data will be projected on three different planes in turn, simplifying the three-dimensional information into a two-dimensional representation: XY plane projection: obtain the perspective of the two-dimensional distribution of the point cloud on the plane; XZ and YZ plane projection: obtain the projection perspectives in the other two perpendicular directions respectively. This orthogonal projection allows the network to observe the geometric structure of the point cloud from different angles, helping to capture feature information in different directions. At the same time, the added two-dimensional distance map is a way to represent the depth of the point cloud, recording the distance from each point in the point cloud to the sensor. This distance map can provide distance information from the object to the LiDAR, helping the network to better understand the depth relationship in the point cloud.

[0097] In the spatial mixing module, two 2D depthwise convolutions and pointwise convolutions are used, which only operate in the spatial dimension without changing the channel information, and a ReLU activation function is inserted to increase nonlinearity. This is followed by a 1D grouped convolution, which operates in the channel dimension to help information interaction between different channels.

[0098] At the same time, in the channel mixing module, features are mainly mixed in the channel dimension rather than the spatial dimension, which helps the information flow and feature fusion between different channels. Batch normalization and jump connection are added at the same time: batch normalization is first performed on the input of the spatial mixing module / channel mixing module to help the model to train stably between different batches; finally, the output of the spatial mixing module / channel mixing module is added to the input (i.e., element-by-element addition) through jump connection to retain input information, help information flow and avoid the gradient vanishing problem. This structure combines feature mixing in the spatial and channel dimensions, thereby improving the network's ability to extract and fuse different types of features.

[0099] The last stage is the segmentation head stage, which includes a one-dimensional convolution and a Softmax layer. The merged point cloud output from the sharing stage undergoes a one-dimensional convolution, and then the Softmax layer is used to infer the final label. The output label is the classification result of each point in the point cloud, that is, the category to which each point is assigned. For point cloud segmentation tasks, the label usually indicates the category of each point, helping to identify and distinguish different parts or objects in the point cloud.

[0100] The deep learning architecture PCformer is implemented in Python and accelerated using C++ and CUDA. After training with the data set, the trained point cloud segmentation model is obtained, which is used to detect and classify roadblock information in real time in autonomous driving scenarios. During the training process, the Adam W optimizer is used to train the model, with an initial learning rate of 0.0001 and a weight decay of 0.0125; a poly learning strategy with a power of 0.9 is used to reduce the learning rate. When the training model reaches convergence, that is, the model training process converges at the 45th training cycle.

[0101] Embodiment 2:

[0102] In this embodiment, a point cloud real-time semantic segmentation automatic driving roadblock detection system includes:

[0103] Point cloud data acquisition module, used to obtain three-dimensional point cloud data of the scene through LiDAR;

[0104] Point cloud data integration module, which is used to integrate the three-dimensional coordinates of the Cartesian coordinate system of the point in the point cloud data, the LiDAR intensity value and the Euclidean distance between the LiDAR emission point and the target object into a point data format;

[0105] Point cloud segmentation model, which is used to predict scene labels for input point clouds and obtain segmentation maps. Each pixel in the segmentation map has a corresponding label and confidence score.

[0106] The label post-processing module is used to perform conflict resolution and boundary smoothing on the results of the point cloud segmentation model to improve the segmentation quality. The specific process is:

[0107] 1) Conflict resolution: traverse the segmentation results of the point cloud segmentation model, check whether there are different parts of the same object, define the different segmentation maps obtained by the segmentation results as regions, and the labeled objects identified in the region as parts. There are the same labeled objects separated by adjacent regions, and different parts of the same labeled objects separated in adjacent regions; set the distance threshold. When the pixel labels in two regions are the same but belong to different parts and the distance is less than the distance threshold, it is considered a conflict;

[0108] For the detected conflicts, a confidence threshold is set for each type of label. The confidence scores of all pixels in the area are averaged to obtain the confidence score of the area. The confidence score of each area is obtained. If the confidence score of the current area is higher than the confidence threshold of the current label, the labels of the pixel images segmented from the area are retained and merged. If not, the label is deleted from the area.

[0109] At the same time, the contour extraction algorithm is used to identify the boundaries of the object and obtain the optimized image after updating the label;

[0110] 2) Boundary smoothing: For the optimized graph after updating the label, each pixel is regarded as a node in the graph, and the adjacent pixels establish edges to build a graph model; define the data item energy function E data And the smoothing term energy function E smooth ;

[0111] The data term energy function is used to reflect the labels of the original segmentation: ,

[0112] in is the true label (after conflict optimization), is the initial segmentation label (from the segmentation model), is the indicator function;

[0113] The smoothness energy function is used to encourage the consistency of adjacent pixel labels: ,

[0114] Where N2 is the set of adjacent pixel pairs, are pairs of adjacent pixels.

[0115] Then set the learning rate and the maximum number of iterations, and use iterative optimization to minimize the data term energy function E dataAnd the smoothing term energy function E smooth , and obtain the final optimized smoothing graph.

[0116] The target recognition and classification module is used to identify different types of objects based on the results of the label post-processing module, assign a category label to each identified object, extract the geometric features of the object to calculate the volume of the object, and the shape features of the object to calculate the aspect ratio and roundness. The segmentation results are aggregated to calculate the geometric center C and bounding box of each object, and the vehicle's position coordinates V are obtained through the GPS sensor. The position R (R) of the object relative to the vehicle is determined according to R = C - V. x , R y , R z );

[0117] If R x > 0, the object is in front of the vehicle. If R x < 0, the object is behind the vehicle; if R y > 0, the object is on the left side of the vehicle; if R y <0, the object is on the right side of the vehicle; if R z < 0, the object is located under the vehicle; if R z > 0, the object is above the vehicle;

[0118] The target tracking module continuously monitors the state and movement of objects;

[0119] The early warning feedback module is used to monitor the surrounding obstacles in real time through collision risk assessment and early warning triggering, and quickly issue an alarm when potential danger occurs.

[0120] In the present invention, a point cloud segmentation model is used to divide the point cloud into multiple adjacent but non-overlapping sub-regions, and each region corresponds to a part of the point cloud. Each region contains: the coordinate information of the point; the corresponding label; the confidence score of the label. The same labeled object may be divided into multiple discontinuous parts in different regions. It is necessary to determine whether these parts belong to the same object and merge them when necessary. Set the distance threshold = 5 pixels. When the points of the same label in two adjacent regions are divided into multiple parts, but actually should belong to the same object and the distance is less than the distance threshold, it is considered a conflict. For the detected conflict, set the confidence threshold = 0.7, obtain the confidence score of each region (the confidence scores of all pixels are averaged, and the confidence threshold is set for one label). If the confidence score of the current region is higher than the confidence threshold of the current label, the label of the region is retained and merged. If it is not higher, the label is deleted.

[0121] Embodiment 3:

[0122] In the label post-processing module of this embodiment, the contour extraction algorithm is the QuickHull convex hull algorithm, which is used to identify the boundary of the object: first, the boundary points in the point cloud data are extracted, and these points will be used as the input of the contour extraction algorithm. The QuickHull convex hull algorithm is used to process them to obtain multiple convex hull point sets, which are then spliced ​​into a global convex hull through the convex hull point sets, and the QuickHull convex hull algorithm is used again to merge the convex hulls.

[0123] The QuickHull convex hull algorithm is mainly carried out in the following four steps:

[0124] ① Find the smallest rectangle (maximum x and y coordinates) that contains all the points;

[0125] ② Starting from the two diagonal points of the smallest rectangle, divide the point set into two parts, left and right, and select the two leftmost and rightmost points. Based on the straight line formed by these two points, divide the remaining points into two groups: the left point set: points on the left side of the straight line; the right point set: points on the right side of the straight line;

[0126] ③ For each part of the point set, find the point farthest from the reference line (called the farthest point), which must be part of the convex hull. Use the farthest point to further divide the point set into two new subsets, namely the left point set from the starting point of the reference line to the farthest point, and the right point set from the farthest point to the end point of the reference line. Repeat this process until the subset is empty and no points remain;

[0127] ④ Collect all boundary points to form a convex hull.

[0128] Boundary smoothing: All boundary points are obtained after being processed by the contour extraction algorithm. For the optimized graph after updating the labels, each pixel is regarded as a node in the graph, and edges are established between adjacent pixels to construct a graph model.

[0129] Define two energy functions, one data term energy function (reflecting the labels of the original segmentation): ,in is the true label (after conflict optimization), is the initial segmentation label (from the segmentation model), is the indicator function; a smoothing term energy function (encouraging consistency of adjacent pixel labels) , where N2 is the set of adjacent pixel pairs.

[0130] Then set max_iterations=100 and learning rate learning_rate=0.01, use iterative optimization to minimize the two energy functions, and obtain the final optimized smoothing graph, that is, the optimized label area graph.

[0131] By analyzing the label information output by the point cloud segmentation model, different object types in the image are identified, and the pixels of each object are assigned to specific category labels, such as pedestrians, vehicles, obstacles, etc.

[0132] The extracted features are trained using a classifier support vector machine to assign a corresponding class label to each identified object, which can be based on the object's appearance characteristics, shape, size, and other contextual information. The following steps are used for further analysis and decision support.

[0133] ① Geometric feature extraction: By integrating the pixel area of ​​an object, the volume of the object can be calculated.

[0134]

[0135] Among them A i is the pixel area of ​​each object, is the height of the i-th part of the object, and N3 is the total number of parts into which the object is divided.

[0136] ②Analyze the shape characteristics of the object and calculate the aspect ratio and roundness.

[0137] Aspect Ratio = , roundness =

[0138] Where A is the area of ​​the object and p is the perimeter. The result of the label post-processing module is the point cloud of the three-dimensional object, which is projected onto the plane. Then the minAreaRect function of the OpenCV library in Python is called to find the minimum enclosing rectangle containing the object and extract the width and height: the two sides of the enclosing rectangle are the width and height respectively, the longer side is used as the width and the shorter side is used as the height.

[0139] ③ By aggregating the segmentation results, calculate the geometric center, bounding box and other features of each object.

[0140] Geometric Center: Calculate the geometric center of all points of each object:

[0141]

[0142] Where C is the geometric center coordinate (C x ,C y ,C z ), N4 is the number of points in the point cloud of an object, P i =(x i ,y i ,z i ) are the coordinates of the ith point.

[0143] Bounding box: Determine the boundaries of the object through the minimum / maximum coordinates, and obtain the width, height, and depth of the bounding box.

[0144] Minimum coordinates:

[0145] Maximum coordinates:

[0146] Bounding box dimensions:

[0147] Relative position calculation: Determine the position of an object relative to the vehicle (such as the front, left, right), providing a basis for subsequent obstacle avoidance decisions.

[0148]

[0149] Where R is the relative position coordinate (R x ,R y ,R z ), C is the geometric center of the object, V is the position coordinate of the vehicle (V x ,V y ,V z ), obtained through the GPS sensor.

[0150] Based on the relative position, the following conditions can be used to determine in which direction the object is in the vehicle:

[0151] Front: If R x > 0.

[0152] Rear: If R x < 0.

[0153] Left: If R y > 0.

[0154] Right side: If R y < 0.

[0155] Above: If R z > 0.

[0156] Below: If R z < 0.

[0157] In this embodiment, the target tracking module helps the system to continuously monitor the state and movement of objects, especially in complex traffic environments. The specific process is:

[0158] 1. Apply the tracking algorithm Kalman filter to track the identified objects in real time.

[0159] 2. Real-time monitoring: Combine multiple sensors (such as lidar, camera, ultrasonic sensor, where the sensor installation method is based on existing technology) to perceive the environment and enhance detection accuracy. Update point cloud data regularly to capture new obstacles or changes. By continuously acquiring video frames, the system can update the position and motion status of objects in real time.

[0160] 3. Target matching: During the tracking process, the objects in the current frame need to be matched with the objects in the previous frame. This process uses the object features extracted by the target recognition and classification module to describe the uniqueness of the object, which helps to identify and distinguish different objects. At the same time, the Euclidean distance is calculated to evaluate the similarity between the objects in the current frame and the previous frame. The smaller the distance, the greater the possibility of matching. In addition, in multi-object tracking, the Hungarian algorithm is used to solve the allocation problem in data association to ensure the identity consistency of each object in different frames.

[0161] The process of early warning judgment by the early warning feedback module in this embodiment is:

[0162] Calculate the distance between the vehicle and surrounding obstacles in real time to assess safety.

[0163] Distance calculation:

[0164] Assume the coordinates of the center point of the vehicle are P vehicle (x v ,y v ,z v ), the geometric center coordinates of the obstacle are P obstacle (x o ,y o ,z o ), the Euclidean distance D between the center point of the vehicle and the geometric center of the obstacle is calculated according to the following formula.

[0165]

[0166] Set the safety distance threshold D safe , safety distance threshold D safe The specific value of is adjusted according to the complexity of the environment and the speed of the vehicle. Multiple safety distance thresholds can also be defined to adapt to different scenarios.

[0167] When the speed is slow, D safe, low =5 m

[0168] When the speed is faster, D safe, high =10 meters.

[0169] Relative speed calculation:

[0170] Let the vehicle speed be V v (Unit: m / s), the obstacle speed is Vo , then the relative speed between the vehicle and the obstacle is V relative for:

[0171]

[0172] Combine vehicle speed and distance information to determine potential collision risk, that is, based on the current speed V v , the moving speed of the obstacle V o and distance D, to determine the collision risk level, which can be set as:

[0173] Low risk: when D>D safe, high .

[0174] Medium risk: when D≤D safe, high And D>D safe, low .

[0175] High risk: when D≤D safe, low And V relative >0 (i.e., the two are close).

[0176] In the case of potential danger, early warnings are issued in a timely manner. When a warning is triggered, different warning messages are triggered according to the risk level to notify the driver or control system.

[0177] Warning triggers include sound and light prompts, dashboard alarms, and vibration feedback, among which:

[0178] Sound and light prompts: flashing lights;

[0179] Instrument panel alert: Displays warning messages on the display;

[0180] Vibration feedback: The seat vibrates to give a prompt.

[0181] When D>D safe, high When , it is a low risk and no warning is triggered;

[0182] When D≤D safe, high And D>D safe, low When a medium risk warning is triggered, the light flashes and a warning message is displayed on the display screen;

[0183] When D≤D safe, low And V relative >0 (i.e. the two are close), a high-risk warning is triggered. Not only will the light flash, a warning message will be displayed on the display, but the seat will also vibrate as a prompt.

[0184] This embodiment uses the point cloud segmentation model in Example 1 to make a fair comparison with the current advanced point cloud segmentation method, and uses the data set obtained by the point cloud data acquisition module for training and evaluation. The present invention uses the category label intersection over union (IoU) and the average score of all category labels (mIoU) as performance metrics. In the point cloud embedding stage, it is necessary to build a KD tree on the input point cloud. In this embodiment, GPU acceleration is used to transfer the KD tree calculation to the GPU. Using GPU optimization, the total time required to build and query the tree is sharply reduced from about 160 milliseconds to only 15 milliseconds. At the same time, in order to verify the real-time performance of this method, the present invention analyzes the number of layers of the network structure of the combination of the spatial mixing module and the channel mixing module. Through the analysis of the number of network layers, it is experimentally found that the most effective results are obtained when the number of network layers L=12 of the combination of the spatial mixing module and the channel mixing module is selected. The experimental test results obtained on the data set of this article are shown in Tables 1 and 2 below;

[0185]

[0186]

[0187] Embodiment 4:

[0188] The system of this embodiment (see Figure 3 )include:

[0189] Point cloud data acquisition module: Use LiDAR to obtain scene point cloud data, mainly to obtain the ten types of scene data required by this article, and label each type of data;

[0190] Point cloud data integration module: convert the corresponding data into the required form and integrate it into the required format to prepare for the next step;

[0191] Point cloud data preprocessing module: voxelizes and crops the acquired point cloud data to remove noise and redundant data, thus improving efficiency and accuracy for subsequent steps;

[0192] Point cloud real-time semantic segmentation module: connected with the point cloud data preprocessing module to obtain the processed point cloud data for semantic segmentation, which is used for subsequent real-time autonomous driving roadblock detection;

[0193] Label post-processing module: performs secondary processing on the labels obtained after segmentation to obtain higher quality labels;

[0194] Target recognition and classification module: Use the label information processed by the label post-processing module to identify different types of objects, assign a category label to each identified object, extract the geometric features of the object, and determine the relative position of the object to prepare for the next step;

[0195] Object tracking module: applies tracking algorithms to track the identified objects in real time, helping the system to continuously monitor the objects and match the objects in the current frame with those in the previous frame during the tracking process to maintain the identity and movement trajectory of the objects;

[0196] Early warning feedback module: connected to the target tracking module, the target tracking module updates the position and motion status of the object in real time, promptly warns and feeds back alarm information to the background system, and the background system automatically issues an alarm and prompts the driver or control system to take measures.

[0197] The point cloud real-time semantic segmentation automatic driving roadblock detection method of the present invention is highly innovative and practical, and can accurately identify and detect roadblocks in complex driving environments. It cleverly utilizes the local feature extraction technology in the point cloud data embedding process, as well as the hybrid projection of three-dimensional point cloud and range image, thereby significantly improving the accuracy and efficiency of detection. In practical applications, the system can not only analyze the surrounding environment in real time and identify potential roadblocks, but also provide timely feedback to help the automatic driving system respond quickly to ensure driving safety. It takes into account the needs of various working environments and is suitable for diverse places such as urban streets, highways, and complex traffic scenes.

[0198] The present invention significantly reduces the operating time of the system, realizes rapid real-time detection of roadblocks, improves the response speed and safety of autonomous driving vehicles, provides a new solution for the development of autonomous driving technology, promotes the progress of the intelligent transportation field, and has wide applicability and important practical significance. At the same time, while improving the roadblock detection capability, the system also provides strong technical support for future autonomous driving safety assurance, showing broad application prospects.

[0199] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A point cloud real-time semantic segmentation automatic driving roadblock detection method, characterized in that: The detection method comprises the following contents: Using LiDAR to obtain three-dimensional point cloud data of the scene, including the three-dimensional coordinates of the point in the Cartesian coordinate system, the LiDAR intensity value and the Euclidean distance between the LiDAR emission point and the target object; and labeling each frame of the collected point cloud data with corresponding labels to obtain a data set; the labels include fence class, bicycle class, bus class, car class, trailer class, truck class, motorcycle class, sidewalk class, grass class, and pedestrian class; Build a point cloud segmentation model: The point cloud segmentation model adopts a deep learning architecture PCformer, which includes a point cloud embedding stage, a sharing stage, and a segmentation head stage; After preprocessing the point cloud data, a point cloud P0∈R is obtained. N1×5 , where N1 is the number of points in the point cloud, and 5 is the dimension of the feature of each point; in the point cloud embedding stage, P0 maps the five features to 128 dimensions through a one-dimensional convolutional layer to obtain a high-dimensional feature point cloud P1; at the same time, a K-dimensional tree is constructed on the point cloud P0, and the K-dimensional tree is used to retrieve the K nearest neighbors of each point and construct the tensor P n , tensor P n Then, the neighboring feature point cloud P is obtained after two two-dimensional convolutions and the rectified linear unit activation function. n ∈R K×N1×5 , then, for the neighboring feature point cloud P n Perform the maximum pooling operation to obtain the output point cloud P2∈R N1×128 , point cloud P2 has the same number of points and feature size as P1; the high-dimensional feature point cloud P1 and point cloud P2 are concatenated to obtain point cloud P3, and then point cloud P3 is remapped to a feature size of 128 through one-dimensional convolution. The point cloud after one-dimensional convolution mapping is represented as P e ∈R N1×128 ; In the sharing phase, P e As input, the point cloud P is processed through a series of spatial mixing modules and channel mixing modules. e Processing is performed to obtain a merged point cloud; the number of combinations of spatial mixing modules and channel mixing modules in the shared stage is four, and each spatial mixing module processes data on different planes; In the segmentation head stage, including one-dimensional convolution and Softmax layer, the merged point cloud output in the sharing stage undergoes a one-dimensional convolution, and then the Softmax layer is used to infer the final label. The output label is the classification result of each point in the point cloud. The deep learning architecture PCformer is implemented in Python and accelerated using C++ and CUDA. After training with the dataset, a trained point cloud segmentation model is obtained, which is used to detect and classify roadblock information in real time in autonomous driving scenarios. Four spatial mixing modules map 3D point clouds into different 2D projections. Each spatial mixing module includes a batch normalization module, a mapping module, a two-dimensional deep convolution DW Conv2D, a ReLU activation function, a two-dimensional point-by-point convolution PW Conv2D, a remapping module, and a one-dimensional grouped convolution Gp Conv1D. The four mapping modules in the four spatial mixing modules map the input point cloud into the orthogonal projection of plane XY, the orthogonal projection of plane XZ, the orthogonal projection of plane YZ and a two-dimensional distance map in sequence; the remapping module is used to restore the 2D projection mapping to a 3D point cloud; The channel mixing module includes a batch normalization module, two one-dimensional depthwise convolutions and one-dimensional pointwise convolutions separated by ReLU activation functions, and a one-dimensional grouped convolution; The output of the spatial mixing module is jump-connected to the input as the input of the channel mixing module, and the input and output of the channel mixing module are jump-connected as the input of the next level.

2. The method according to claim 1, characterized in that The process of constructing the K-dimensional tree is as follows: for each point in the point cloud P0, find its K neighbors, then stack the 5-dimensional features of these neighbors to form a neighbor feature set, and concentrate the neighbor feature sets of all points in the tensor P n middle.

3. The method according to claim 1, characterized in that The process of preprocessing point cloud data is: converting continuous three-dimensional point cloud data into discrete voxels; cropping the point cloud data to the LiDAR field of view, retaining only the points that the LiDAR can see, and discarding the points outside the LiDAR field of view.

4. A point cloud real-time semantic segmentation automatic driving roadblock detection system, characterized in that: The system comprises: Point cloud data acquisition module, used to obtain three-dimensional point cloud data of the scene through LiDAR; Point cloud data integration module, used to integrate the three-dimensional coordinates of points in the point cloud data, LiDAR intensity values, and the Euclidean distance between the LiDAR emission point and the target object into a point data format; Point cloud segmentation model, which is used to predict scene labels for input point clouds and obtain segmentation maps. Each pixel in the segmentation map has a corresponding label and confidence score. The label post-processing module is used to perform conflict resolution and boundary smoothing on the results of the point cloud segmentation model. The specific process is: 1) Conflict resolution: traverse the segmentation results of the point cloud segmentation model, check whether there are different parts of the same object, define the different segmentation images obtained by the segmentation results as regions, and the labeled objects identified in the region as parts. There are the same labeled objects separated by adjacent regions, and different parts of the same labeled objects separated in adjacent regions; set a distance threshold. When the pixel labels in two regions are the same but belong to different parts and the distance is less than the distance threshold, it is considered a conflict; for the detected conflict, set a confidence threshold for each type of label, and average the confidence scores of all pixels in the region as the confidence score of the region, and obtain the confidence score of each region. If the confidence score of the current region is higher than the confidence threshold of the current label, the label of the pixel image segmented from the region is retained and merged. If it is not higher, delete the label in the region; At the same time, the contour extraction algorithm is used to identify the boundaries of the object and obtain the optimized image after updating the label; 2) Boundary smoothing: For the optimized graph after updating the label, each pixel is regarded as a node in the graph, and the adjacent pixels establish edges to build a graph model; define the data item energy function E data And the smoothing term energy function E smooth ; Data item energy function E data : where y p is the conflict-optimized true label, is the initial segmentation label, δ is the indicator function; Smooth term energy function E smooth : Where N2 is the set of adjacent pixel pairs, (p1, p2) is an adjacent pixel pair; Then set the learning rate and the maximum number of iterations, and use iterative optimization to minimize the data term energy function E data And the smoothing term energy function E smooth , obtain the final optimized smoothing graph; The target recognition and classification module is used to identify different types of objects based on the results of the label post-processing module, assign a category label to each identified object, extract the geometric features of the object to calculate the volume of the object, extract the shape features of the object to calculate the aspect ratio and roundness, aggregate the segmentation results to calculate the geometric center C and bounding box of each object, obtain the vehicle's position coordinates V through the GPS sensor, and determine the object's relative position R (R x , R y , R z ); If R x >0, the object is in front of the vehicle. If R x <0, the object is behind the vehicle; if R y >0, the object is on the left side of the vehicle; if R y <0, the object is on the right side of the vehicle; if R z <0, the object is located under the vehicle; if R z >0, the object is above the vehicle; The target tracking module continuously monitors the state and movement of objects; The early warning feedback module is used to monitor the surrounding obstacles in real time through collision risk assessment and early warning triggering, and quickly issue an alarm when potential danger occurs.

5. The system according to claim 4, characterized in that The process of collision risk assessment is: Using geometric information, calculate the Euclidean distance D between the center point of the vehicle and the geometric center of the obstacle. Let the coordinates of the center point of the vehicle be P vehicle (x v ,y v ,z v ), the geometric center coordinates of the obstacle are P obstacle (x o ,y o ,z o ); Let the vehicle speed be V v , in meters per second, the obstacle speed is V o , then the relative speed between the vehicle and the obstacle is V relative for: Vrelative=Vv-Vo Set the slow safety distance threshold D safe,low and fast safety distance threshold D safe,high , according to the relative speed V relative And the Euclidean distance D, determine the collision risk level: When D>D safe,high is low risk; when D≤D safe,high And D>D safe,low For medium risk; when D≤D safe,low And V relative >0 indicates high risk.

6. The system according to claim 5, characterized in that Warning triggers include sound and light prompts, dashboard alarms and vibration feedback: When D>D safe,high When , it is a low risk and no warning is triggered; When D≤D safe,high And D>D safe,low When the medium risk warning is triggered, the light flashes and the medium risk warning message is displayed on the display screen; When D≤D safe,low And V relative When >0, a high-risk warning is triggered. Not only will the light flash, a high-risk warning message will be displayed on the display, but the seat will also vibrate as a prompt.

7. The system according to claim 4, characterized in that according to Calculate the volume of an object, Among them, A i is the pixel area of ​​each object part, h i is the height of the i-th part of the object, N3 is the total number of parts into which the object is divided; The aspect ratio and roundness are: Where A is the area of ​​the object, p is the perimeter; the width and height refer to the width and height of the smallest circumscribed rectangle containing the object, respectively, with the long side of the smallest circumscribed rectangle as the width and the short side as the height; Compute the geometric center of all points of each object: Where C is the geometric center coordinate (C x ,C y ,C z ); N4 is the number of points in the point cloud of an object; P i =(x i ,y i ,z i ) is the coordinate of the i-th point; Bounding box: Determine the boundary of the object through the minimum / maximum coordinates, and obtain the width, height, and depth of the bounding box; Minimum coordinates: Maximum coordinates: Bounding box dimensions:

8. The system according to claim 4, characterized in that The contour extraction algorithm adopts the QuickHull convex hull algorithm, and uses the QuickHull convex hull algorithm for processing to obtain multiple convex hull point sets, which are spliced ​​into a global convex hull through the convex hull point sets, and the QuickHull convex hull algorithm is used again to merge the convex hulls.

Images