Method and apparatus for deep learning based 3D lidar point cloud semantic segmentation

Abstract

A deep learning-based 3D lidar point cloud semantic segmentation method includes building a lightweight deep fully convolutional neural network, preprocessing the lidar atomic point cloud, converting 3D spatial features to 2D spatial features, training the lightweight deep fully convolutional neural network, 2D point cloud semantic segmentation, post-processing and optimization of the 2D semantic labels, and restoring the 3D semantic labels. The present invention also includes a deep learning-based 3D lidar point cloud semantic segmentation device. The present invention uses polar coordinate system transformation to achieve feature dimensionality reduction and utilizes the cross-stage feature fusion characteristics of the deep learning-based lightweight fully convolutional neural network CSP RangeNet. This not only ensures the accuracy of the segmentation results, but also reduces space consumption and the computational complexity of the network feature extraction stage, significantly improving the segmentation speed. The present invention is applicable to 3D lidar point cloud semantic segmentation, which has relatively high real-time requirements.

Description

[Technical Field]

[0001] The present invention relates to the technical field of computer vision, and more particularly to a semantic segmentation method and apparatus, device and medium based on deep learning and lidar point clouds, which are used for high-precision real-time semantic segmentation of 3D point clouds in outdoor large-scale scenes. [Background technology]

[0002] In autonomous vehicle environment sensing technology, vehicle-mounted lidar has extremely high angle and range resolution, enabling it to capture 360-degree full coverage of environmental object information in large-scale scenes, generating a large amount of high-precision 3D point clouds. As a fundamental technology in 3D point cloud processing, semantic segmentation technology is the basis for various applications such as 3D scene reconstruction, scene understanding, and target detection. It is a hot topic in the field of agent environment sensing and has attracted the attention of many researchers.

[0003] In recent years, deep learning has achieved significant results in the field of computer vision, such as target identification, detection, and semantic segmentation, and many deep learning-based algorithms for semantic segmentation of 3D point clouds of large-scale outdoor scenes have been presented one after another.

[0004] Semantic segmentation based on source point clouds, such as PointNet and SPGraph, performs calculations based on source point sets and can retain complete feature information. However, due to problems such as irregular memory access and neighborhood search caused by disorder during the calculation process, the calculation efficiency is extremely low. Therefore, it is only applicable to indoor scenes with relatively high accuracy requirements and relatively low real-time requirements, and is not applicable to large-scale road scenes with high real-time requirements.

[0005] Voxel-based semantic segmentation methods, such as Kd-Net and MSNet, use the average features of the original point cloud to convert them into single-voxel features, constructing a 3D voxel mesh similar to a 2D pixel map. By constructing a computational diagram, computational and memory resources are adaptively allocated based on the density of the point cloud, resulting in effective segmentation of the point cloud space. The segmentation results in features extracted layer by layer from bottom to top. Due to relatively low correlation between channels, segmentation accuracy is low. At the same time, this type of method is significantly affected by voxel resolution. While high-resolution voxels retain more information, their computational cost and memory consumption are very high, making it difficult to train models based on them. Low-resolution voxels lose a significant amount of information, resulting in quantization errors.

[0006] As described above, these methods still have problems in practical application, such as slow computation speed and low estimation accuracy. At the same time, the memory and performance of most on-board computing devices cannot support the complex structure and huge computational load, making them unable to meet the requirements of real-time semantic segmentation of large-scale outdoor scenes. Summary of the Invention

[0007] The present invention overcomes the shortcomings and deficiencies of the prior art and presents a deep learning-based 3D LIDAR point cloud semantic segmentation method and apparatus, which can achieve more efficient and accurate semantic segmentation for large-scale outdoor scenes. The present invention provides a method for 3D lidar point cloud semantic segmentation based on deep learning, which includes the following steps: Step 1: Build a lightweight deep fully convolutional neural network. We build a fully convolutional neural network for training, which consists of three parts: an encoder, a decoder, and a classifier. Step 2: Perform feature transformation on the original lidar point cloud. The original 3D point cloud dataset is obtained from the lidar, and then undergoes feature transformation and joining to reduce it to four feature dimensions. Step 3: Convert the 3D spatial features into 2D spatial features and perform preprocessing. A structured 4-channel distance feature map is generated from the chaotic features using a spherical projection method, and a single-channel 2D semantic image is generated from the sequence labels and converted into a format that matches the format required for preprocessing data. Then, preprocessing is performed on the data, and the preprocessed data samples are divided into a training set and a test set. Step 4: Train a lightweight deep fully convolutional neural network. Using the point cloud feature maps in the training set as training samples and the point cloud semantic map as ground truth labels, we train a deep fully convolutional semantic segmentation neural network using backpropagation and random gradient descent, and finally obtain optimized model parameters by minimizing the loss function. Step 5: Perform semantic segmentation on the 2D point cloud. The original point cloud is input into the 2D semantic segmentation frame to obtain 2D spatial semantic labels. Step 6: Optimize and post-process the 2D semantic labels. Using a sliding window voting mechanism based on a 2D matrix, we exploit the consistency of point cloud density within a single object and the invariance of rotation and translation to comprehensively correct labeling errors caused by the small amount of single-class point cloud frame data, sparse point clouds due to long-distance sampling, and ambiguous semantic boundaries. Step 7: Recover 3D semantic labels. The label results after 2D semantic segmentation are demapped into the original 3D point cloud semantic space to obtain a complete point cloud frame with 3D semantic labels.

[0008] Furthermore, the lightweight deep fully convolutional network described in step 1 includes an input layer, a CSPDarknet encoder consisting of a cross-stage partial feature fusion strategy (CSP) and a residual structure, a decoder based on a deconvolution structure, and a classifier for pixel-by-pixel classification, forming a point cloud segmentation network, abbreviated as CSP RangeNet. The input is a preprocessed multi-channel 2D point cloud range image, and the output is a 2D semantic label corresponding to each pixel of the range image.

[0009] The above input layer consists of a convolutional layer, a batch normalization layer, and a Mish activation function layer (Conv-BN-M). The Mish activation function is as shown in formula (1-3) below. Mish(x)=x×tanh(ξ(x))(1) TIFF2025542029000002.tif20138(2) ξ(x)=log(1+e x )(3) Where x is the input feature map, tanh(x) is the hyperbolic tangent function, and ξ(x) is an exponential function with base e. The size of the convolution core in the convolution layer is 1x1, the number is 32, and the step length is 1x1. After the input data undergoes feature transformation and preprocessing, it becomes a range feature map with size CxHxW, where C is the number of feature channels (here 4), H is the number of vertical laser beams of the lidar, and W is the horizontal angular resolution of the range feature map.

[0010] The above encoder is a CSPDarknet, which consists of a cross-stage feature fusion layer and a residual network, and is divided into five sub-encoders. Each encoder consists of a downsampling layer (DS), a cross-stage feature fusion residual layer (CSP-Res), and a dropout layer (D). Of these, each downsampling layer consists of a convolutional layer-batch normalization layer-Mish activation function layer (C(3*3 / 2)-BN-M). The size of the convolutional core is 3x3, the step length is 1x2, and the numbers in the five sub-encoders are [64, 128, 256, 512, 1024], respectively. After passing through this layer, the distance feature map remains unchanged in H dimension, halves in W dimension, and doubles in C dimension, which accelerates the subsequent feature extraction calculation speed. The feature extraction structure of each sub-encoder consists of a cross-stage feature fusion residual layer (CSP-Res), differing in the number of residual blocks. The five sub-encoders have the following numbers, respectively: [1, 2, 8, 8, 4]. The cross-stage feature fusion residual layers are [CSP-Res1, CSP-Res2, CSP-Res8, CSP-Res8, CSP-Res4]. Deep features can be extracted from the downsampled feature map through these layers. The dropout layer (D) drops out the extracted deep feature map with probability P.

[0011] The above CSPDarknet encoder includes a schematic diagram of five CSP-Res module structures, a cross-stage feature fusion layer (Partial Transition Layer), a hierarchical feature fusion mechanism, adopting the concept of pre-fusion and post-fusion, using a gradient flow blocking strategy to maximize the difference in gradient combinations, preventing the residual network and other convolutional layers from learning overlapping gradient information, and theoretically improving the learning ability of the network. The specific steps are as follows: (11) 2C*H*W feature maps are input and passed through the left branch convolutional layer (c(C*1*1)), where the convolutional core size is 1×1, the step length is 1×1, and the number is C, to obtain C*H*W feature maps. (12) The 2C*H*W feature map is input and passed through the deep feature extraction layer in the right branch, which includes two convolutional layers (c(c*1*1)) and one stacked residual network (Res*N). The specific flow is as follows: (2a) 2C*H*W feature maps are input to the first convolutional layer c (c*1*1), where the size of the convolutional core is 1×1, the step length is 1×1, and the number of elements is C. (2b) The result of step (2a) is input into the second layer residual network structure (Res*N), which consists of N stacked residual blocks. The specific calculation flow of each residual block is as follows: (2b-1) The result of step (2a) is input to the first convolutional layer (c(C*1*1)) of the residual block, where the size of the convolutional core is 1×1, the step length is 1×1, and the number is C. (2b-2) The result of step (2b-1) is input to the second convolutional layer (c(C*3*3)) of the residual block, where the size of the convolutional core is 3×3, the step length is 1×1, and the number is C. (2b-3) Add the result of step (2b-2) to the result of step (2a). (2b-4) The results of (2b-3) are input to the subsequent N-1 residual blocks for successive calculations. (13) The result of (2b-4) and the result of step (11) are concatenated on the first channel (Cat). (14) The result of step (13) is input to the last convolutional layer (c(2C*1*1)) for feature fusion, where the size of the convolutional core is 1×1, the step length is 1×1, and the number is 2C. The size of the final output feature map is 2C*H*W.

[0012] The decoder consists of five sub-decoders, each consisting of an upsampling module (US) and a residual block (Res). The upsampling module has a deconvolutional structure (ConvTransposed2d(4*4 / 2)), with a convolutional core size of 4x4 and a step length of 2. The five sub-decoders are arranged in the order [1024, 512, 256, 128, 64]. It upsamples the features extracted by the backbone encoder. It restores the original feature map to a resolution of HxW by increasing the W dimension by 32 times, and then refines the result by adding one residual block (Res). At the same time, after each upsampling, a jump connection is added between the encoder outputs of different resolutions. The decoder output is added to the corresponding encoder output to recover some of the high-frequency edge information lost during the downsampling process. A dropout layer (D) is added to the last sub-decoder, which drops out the shallow feature map after five decoding passes with a probability of P. Here, P is 0.05.

[0013] The classifier part above consists of a convolutional layer (Conv(N*3*3 / 1)) and an exponential normalization layer (Softmax). In the convolutional layer, the size of the convolutional core is 3x3, the step length is 1, and the number is N, where N is the number of 2D semantic labels. In the exponential normalization layer (Softmax), dim=0, which means that the first dimension is normalized and the probability distribution of 2D semantic classification for each pixel is output.

[0014] The original point cloud feature transformation described in step 2 uses a coordinate system transformation method to generate four-dimensional features (Ψ, θ, d, r) as new features, and the specific operation steps are as follows: (21) Analyze the lidar data to obtain a raw lidar point cloud frame, each of which has a size of N × 4, where N is the number of points in each frame, and 4 is the characteristic dimension of each point, i.e., [x, y, z, r], where (x, y, z) is the spatial position of the point cloud in the 3D Cartesian coordinate system, and r is the reflectance intensity of the surface material of the object where the point cloud is located. (22) The (x, y, z) in the original LIDAR point cloud frame is transformed into a polar coordinate system to obtain the spatial position representation (Ψ, θ, d) in the polar coordinate system, where Ψ is the horizontal deflection angle in the polar coordinate system, θ is the vertical deflection angle in the polar coordinate system, and d is the distance from the point to the origin of the coordinate system. The polar coordinate system transformation is as follows: TIFF2025542029000003.tif12138(4) TIFF2025542029000004.tif16138(5) TIFF2025542029000005.tif16138(6) The obtained (Ψ,θ,d) is combined with r to replace the original four-dimensional feature, and a new four-dimensional feature (Ψ,θ,d,r) is obtained. Where, Ψ is the pitch angle of the point cloud in the polar coordinate system, θ is the deflection angle of the point cloud in the polar coordinate system, and d is the distance feature from the point cloud to the origin of the radar coordinate system. After feature transformation, a quadruple feature is constructed, i.e., (Ψ, θ, d, r), where r is the reflectivity of the surface material of the object where the point cloud is located.

[0015] The three-dimensional spatial features described in step 3 are converted into two-dimensional spatial features to generate a structured four-channel distance feature map. The main steps are as follows: (31) Using the spherical projection method, the spatial position features (x, y, z) of each point in the Cartesian coordinate system are used, combined with the parameters of the onboard lidar itself, to obtain the (h, w) index pair for each point, where h is the column index of the range image and w is the row index of the range image. The spherical projection is as follows: (7-9) f=f down +f up (7) TIFF2025542029000006.tif16138(8) TIFF2025542029000007.tif15138(9) where f is the range of the lidar vertical field of view, and f down is the lower limit of the lidar vertical field of view, f up is the lidar vertical field of view upper limit, H is the vertical resolution of the range feature map, typically the lidar beam, W is the horizontal resolution of the range feature map, TIFF2025542029000008.tif14138 is a downward-boundary function. Based on the size of feature d, the quadruple features and (h, w) of all point clouds in the single frame obtained in step 2 are sorted in descending order. The re-sorted quadruple features are mapped to corresponding 2D distance feature maps using (h, w). These feature maps are then stitched together and normalized to obtain a distance feature map with a size of [4×H×W]. H is the vertical resolution of the distance feature map, and W is the horizontal resolution of the distance feature map. (32) A normalization process is applied to the distance feature map with a size of 4 × H × W to obtain the final feature map. The normalization process is as shown in formula (10) below. TIFF2025542029000009.tif17138(10) Among them, x i is the feature value on each feature channel of the feature map, μ i is the feature mean value on the corresponding feature channel, σ i is the feature variance on the corresponding feature channel, and the method performs normalization on each feature channel respectively to obtain a normalized four-channel distance feature map. (33) Using the (h,w) index pair obtained in (1), the original discrete 3D semantic labels are first mapped to continuous labels using a label mapping table. Then, (h,w) is mapped to a 2D image, resulting in a single-channel 2D semantic image with dimensions H × W. To distinguish between major categories during the data labeling process, the correct label indices are discontinuous, and continuous correct labels are required for the subsequent semantic inference stage. At the same time, for some scenes, no distinct semantic classification scenes emerge. Learning the characteristics of such labels can easily interfere with the model. Therefore, it is necessary to prevent the model from overlearning and interfering with the classification information. Therefore, it is necessary to map such labels to existing labels to achieve label merging and continuation. The label merging process is as follows: label mapped ←index[label origin ] Among them, label mapped is the label after mapping, label origin is the original label, and index is the mapping relation dictionary. The input in this process is the 1×N original point cloud label origin where N is the number of points and n is the number of classifications. class and the mapping relationship dictionary has the following format: map_dict:key∈(0,max(origin label ),value∈(0,n class ) Among them, map_dict is a mapping dictionary, key is the key of the dictionary, value is the value of the dictionary, and origin label is the index value of the original label, and max() is the function that takes the maximum value. The final output result is a single-channel 2D semantic map label mapped is.

[0016] In the lightweight fully convolutional neural network training process described in step 4, the loss function adopted is the weighted cross-entropy loss combined model structure loss, and the warm-up learning rate mechanism in random gradient descent is used to dynamically adjust the learning rate to improve convergence efficiency and accuracy. The loss function is as shown in formula (11-14) below. TIFF2025542029000010.tif18138(11) TIFF2025542029000011.tif22122(12) TIFF2025542029000012.tif20122(13) loss=L1+L2(14) Among them, L1 is the weighted cross-entropy loss, C is the classification label, and w C is the weight of the classification C, N is the number of samples, TIFF2025542029000013.tif12140 is the probability estimate of C for the label after model inference for the training set samples, y C is the true probability that the sample label is C, and f C is the proportion of samples classified as C to all samples, ε is the offset, L2 is the structural loss of the model, α is the proportional coefficient of the loss, w is a one-dimensional vector consisting of all parameters of the model, TIFF2025542029000014.tif12140 is the 2-norm and loss is the total loss of the model.

[0017] In the 2D point cloud semantic segmentation process described in step 5, the original point cloud undergoes preprocessing in steps 2, 3, and 4, and then is input into a lightweight deep fully convolutional neural network for point-by-point inference to obtain 2D spatial semantic labels.

[0018] The sliding window voting mechanism based on the 2D matrix described in step 6 determines the pixel points within the region range based on the distance threshold for each pixel point in the 2D semantic label map obtained in step 5, and votes based on the label, thereby performing a comprehensive modification of the 2D semantic label. Since the operations on each point in the 2D matrix are all independent, the operation process can be accelerated by using a graphics processing unit (GPU) for parallel computing.

[0019] In the 3D semantic label recovery described in step 7, the label demapping table is used to demap the optimized 2D semantic labels obtained in step 6 into 3D semantic space, and finally obtain a complete point cloud frame with 3D semantic labels.

[0020] A deep learning based 3D lidar point cloud semantic segmentation device, comprising: an acquisition module for acquiring raw point cloud data; an input module for preprocessing the source point cloud data and extracting preprocessed features of each point in the point cloud data by a first convolutional layer; an encoding module for performing feature extraction and downsampling on the input point cloud features using a predetermined number of encoding layers to obtain intermediate feature mappings corresponding to each of the encoding layers, wherein the encoding layers include a downsampling module, a cross-stage feature fusion residual module, and a dropout module; a decoding module for performing feature decoding and upsampling on the intermediate feature mapping using a predetermined number of decoding layers to obtain an upsampling feature mapping corresponding to each of the decoding layers, wherein the decoding layers include an upsampling module and a residual module; a classification module for collectively summarizing the intermediate feature mappings generated by the decoding layer and performing point-wise two-dimensional semantic classification using the holistic features; a mapping module for remapping the two-dimensional semantic labels generated by the classification module into a three-dimensional space to generate three-dimensional semantic labels; and an output module for aggregating the three-dimensional semantic labels for each point output by the mapping module for each frame and outputting a three-dimensional semantic frame. An electronic device and medium including a lidar, a processor, a removable storage medium, a communication part, an on-board computing cluster, a memory, and a computer program stored in and executed by the memory, wherein the lidar is a rotary lidar having a horizontal field of view range of 360°, and the processor, when executing the program, realizes a deep learning-based three-dimensional lidar point cloud semantic segmentation method according to any one of claims 1 to 9, and the on-board computing cluster is a graphics processing unit (GPU) cluster that realizes the training and inference process according to any one of claims 6 to 9.

[0021] The external electronic devices and media include a solid, semi-solid, or mechanical rotating lidar connected to an input / output (I / O) interface that can capture information in the environment to generate raw point cloud data and input it as a subsequent model; a communication section including a network interface card such as a LAN card or modem that can perform communication processing via a network such as the Internet; and a removable storage medium including a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., into which a computer reads a computer program stored therein from the I / O port or writes data frames of semantic segmentation results that are not permanently stored, as required. The internal electronic devices and media described below include a central processing unit (CPU) that can execute various commands and operations by loading programs stored in read-only memory (ROM) into random access memory (RAM), where various programs and data necessary for the device's operation are also stored in the RAM. The on-board computing cluster receives commands from the CPU and data from the ROM to perform large-scale data parallel calculations. In particular, in this embodiment, it is an on-board graphics processing unit (GPU) cluster connected via a high-speed bus (PCI-E) inside the computer, and the CPU, on-board computing cluster, ROM, and RAM are connected to each other via the bus. The I / O interface is also connected to the bus.

[0022] The present invention further includes a storage medium, which may be included in the aforementioned electronic devices and media, or may exist independently and not be integrated into the device. The storage medium stores one or more programs, which are used by one or more processors to execute the deep learning-based 3D lidar point cloud semantic segmentation method described in the present invention. Information storage can be achieved by any method or technology, including persistent and non-persistent, removable and non-removable media. Information may be computer instructions, program modules, or other data, such as source data, result data, etc. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, or other optical or magnetic memories used to store information accessible by a computing device.

[0023] Compared with the prior art, the present invention has the following advantages: (1) Feature dimensionality reduction: In the data preprocessing stage, the 3D Cartesian coordinate system is replaced with a spherical coordinate system to realize the 3D to 2D feature space transformation, and the dimensionality of the input features is reduced, thereby accelerating the calculation speed of the subsequent deep neural network input layer. At the same time, the new features retain a large amount of information from the original features, ensuring the accuracy of segmentation.

[0024] (2) Construction of lightweight semantic segmentation network CSP RangeNet: Using a branch structure and cross-stage feature fusion strategy, we present a lightweight fully convolutional semantic segmentation network CSP RangeNet based on 2D range images. This reduces the number of parameters by 35% compared to the same type of model (RangeNet), further reducing the memory cost of the model and the computational effort in the training and inference processes, thereby realizing point cloud semantic segmentation at the execution frame rate of the lidar (above 10Hz).

[0025] (3) Inference acceleration: The segmentation network uses a fully convolutional structure, and the optimization stage uses a sliding window voting mechanism based on a two-dimensional matrix. Since these calculation methods are all independent, using a GPU to perform parallel calculations can accelerate inference and further improve the execution speed of the method. [Brief explanation of the drawings]

[0026] [Figure 1] FIG. 1 is an overall flowchart of the deep learning-based 3D LIDAR point cloud semantic segmentation method of the present invention. [Figure 2] FIG. 2 is a schematic diagram of the network structure model of the lightweight deep fully convolutional neural network CSP RangeNet of the present invention. [Figure 3]FIG. 3 is a schematic diagram of the CSP-Res module structure in the encoder provided by the method of the present invention. [Figure 4] FIG. 4 is an overall flowchart of the data preprocessing portion provided by the present invention. [Figure 5] FIG. 5 is a structural schematic diagram of a point cloud semantic segmentation device provided by the present invention. [Figure 6] FIG. 6 is a structural schematic diagram of the electronic device and medium provided by the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0027] The drawings are merely illustrative and should not be understood as limitations on the present invention. In order to more appropriately explain the present embodiment, some elements in the drawings are omitted, enlarged, or reduced in size, but do not represent the actual dimensions of the product. Those skilled in the art will understand that some known structures and their descriptions may be omitted in the drawings. The positional relationships shown in the drawings are merely used for illustrative purposes and should not be understood as limitations on the present invention.

[0028] The present invention will be explained in detail below in conjunction with the drawings and examples. FIG. 1 shows an overall flowchart of a deep learning-based 3D LIDAR point cloud semantic segmentation method provided in an embodiment of the present invention, which includes the following steps: Step 1: Building a lightweight deep fully convolutional neural network. We build a fully convolutional neural network for training, which consists of three parts: an encoder, a decoder, and a classifier. Step 2: Lidar original point cloud feature transformation. The original 3D point cloud dataset is obtained from the Lidar, and then it is reduced to four feature dimensions through feature transformation and joining. Step 3: Convert the 3D spatial features into 2D spatial features and perform preprocessing. A structured 4-channel distance feature map is generated from the chaotic features using a spherical projection method, and a single-channel 2D semantic image is generated from the sequence labels and converted into a format that matches the format required for preprocessing data. Then, preprocessing is performed on the data, and the preprocessed data samples are divided into a training set and a test set. Step 4: Training a lightweight deep fully convolutional neural network. Using the point cloud feature maps in the training set as training samples and the point cloud semantic map as ground truth labels, we train a deep fully convolutional semantic segmentation neural network using backpropagation and random gradient descent, and finally obtain optimized model parameters by minimizing the loss function. Step 5: Semantic segmentation of 2D point cloud. The original point cloud is input into the 2D semantic segmentation frame to obtain 2D spatial semantic labels. Step 6: Post-processing and optimization of 2D semantic labels. Using a sliding window voting mechanism based on a 2D matrix, the consistency of point cloud density within a single object and the invariance of rotation and translation are utilized to comprehensively correct labeling errors caused by the small amount of single-class point cloud frame data, the sparseness of point clouds due to long-distance sampling, and the ambiguity of semantic boundaries. Step 7: Recovering 3D semantic labels. The label results after 2D semantic segmentation are demapped into the original 3D point cloud semantic space to obtain a complete point cloud frame with 3D semantic labels.

[0029] In the above step 1, a point cloud segmentation network, abbreviated as CSP RangeNet, is constructed on the constructed lightweight deep fully convolutional network, which includes an input layer, a CSPDarknet encoder consisting of a cross-stage feature fusion strategy and a residual structure, a decoder based on a deconvolution structure, and a classifier for pixel-by-pixel classification. The input is a preprocessed multi-channel 2D point cloud range image, and the output is a 2D semantic label corresponding to each pixel of the range image.

[0030] In the above step 2, the transformation of the original point cloud features involves transforming the position features (x, y, z) in the original Cartesian coordinate space into (Ψ, θ, d) using the polar coordinate system, where Ψ is the pitch angle in the polar coordinate system of the point cloud, θ is the deflection angle in the polar coordinate system of the point cloud, and d is the distance feature from the point cloud to the origin of the radar coordinate system. After the feature transformation, a quadruple feature, i.e., (Ψ, θ, d, r), is formed, where r is the reflective intensity of the surface material of the object where the point cloud is located.

[0031] The conversion of 3D spatial features into 2D spatial features in step 3 uses a spherical projection method, utilizing the spatial location features (x,y,z) of each point in the Cartesian coordinate system, combined with the vehicle-mounted lidar's own parameters to obtain an (h,w) index pair for each point, where h is the column index of the range image and w is the row index of the range image. Based on the size of feature d, the quadruple features and (h,w) of all point clouds in the single frame obtained in step 2 are sorted in descending order. The reordered quadruple features are mapped to corresponding 2D distance feature maps using (h,w), and these feature maps are then stitched together and normalized to obtain a distance feature map with dimensions 4×H×W, where H is the vertical resolution of the distance feature map and W is the horizontal resolution of the distance feature map. For the original discrete 3D semantic labels, a label mapping table is used to first generate continuous labels through label mapping, and then (h,w) is mapped to the 2D image, resulting in a single-channel 2D semantic image with dimensions H×W.

[0032] In the lightweight fully convolutional neural network training process in step 4 above, the loss function adopted is the weighted cross-entropy loss combined model structure loss, and a warm-up learning rate mechanism is adopted in random gradient descent to dynamically adjust the learning rate, thereby improving the convergence efficiency and convergence accuracy.

[0033] Regarding the 2D point cloud semantic segmentation in step 5 above, the original point cloud undergoes preprocessing in steps 2, 3, and 4, and then is input into a lightweight deep fully convolutional neural network for point-by-point inference to obtain 2D spatial semantic labels.

[0034] In the sliding window voting mechanism based on the two-dimensional matrix in step 6 above, for each pixel point in the two-dimensional semantic label map obtained in step 5, a pixel point within the region is determined based on a distance threshold, and voting is performed based on the label, resulting in a comprehensive modification of the two-dimensional semantic label. Because the operations on each point in the two-dimensional matrix are all independent, the operation process can be accelerated by using a graphics processing unit (GPU) for parallel computing.

[0035] For the 3D semantic label recovery in step 7 above, we use the label demapping table to demap the optimized 2D semantic labels obtained in step 6 into 3D semantic space, finally obtaining a complete point cloud frame with 3D semantic labels. Figure 2 shows a schematic diagram of the network structure model of the lightweight deep fully convolutional neural network CSP RangeNet provided in this embodiment of the present invention, which includes a CSP Darknet encoder constructed based on a cross-stage feature fusion strategy and residual structure, a decoder based on a deconvolution structure, and a classifier for pixel-by-pixel classification. The specific configuration is as follows:

[0036] 1. The input layer consists of a convolutional layer, a batch normalization layer, and a Mish activation function layer (Conv-BN-M). The Mish activation function is expressed as the following formula (1-3). Mish(x)=x×tanh(ξ(x))(1) TIFF2025542029000015.tif16140(2) ξ(x)=log(1+e x )(3) Where x is the input feature map, tanh(x) is the hyperbolic tangent function, and ξ(x) is an exponential function with base e. The size of the convolution core in the convolution layer is 1x1, the number is 32, and the step length is 1x1. After the input data undergoes feature transformation and preprocessing, it becomes a range feature map with size CxHxW, where C is the number of feature channels (here 4), H is the number of vertical laser beams of the lidar, and W is the horizontal angular resolution of the range feature map.

[0037] 2. The encoder is CSPDarknet. Its structure consists of a cross-stage feature fusion layer and a residual network, and is divided into five sub-encoders. Each encoder consists of a downsampling layer (DS), a cross-stage feature fusion residual layer (CSP-Res), and a dropout layer (D). Each downsampling layer consists of a convolutional layer - batch normalization layer - Mish activation function layer (C(3*3 / 2)-BN-M), with the convolutional core size of 3x3 and the step length of 1x2. The number of sub-encoders is [64, 128, 256, 512, 1024], respectively. After passing through this layer, the distance feature map remains unchanged in H dimension, halves in W dimension, and doubles in C dimension, thereby accelerating the subsequent feature extraction calculation speed. The feature extraction structure of each sub-encoder consists of a cross-stage feature fusion residual layer (CSP-Res), differing in the number of residual blocks. The five sub-encoders have the following numbers, respectively: [1, 2, 8, 8, 4]. The cross-stage feature fusion residual layers are [CSP-Res1, CSP-Res2, CSP-Res8, CSP-Res8, CSP-Res4]. Deep features are extracted from the downsampled feature maps through these layers. The dropout layer (D) drops out the extracted deep feature maps with a probability P, where P is 0.05.

[0038] 3. The decoder consists of five sub-decoders, each consisting of an upsampling module (US) and a residual block (Res). The upsampling module has a deconvolutional structure (ConvTransposed2d(4*4 / 2)), with a convolutional core size of 4x4 and a step length of 2. The number of sub-decoders is sequentially [1024, 512, 256, 128, 64]. It upsamples the features extracted by the backbone encoder. This upsamples the original feature map to a resolution of HxW by a factor of 32 in the W dimension, and then refines the result by adding a residual block (Res). At the same time, after each upsampling, a jump connection is added between the encoder outputs of different resolutions. This adds the decoder output to the corresponding encoder output to recover some of the high-frequency edge information lost during the downsampling process. A dropout layer (D) is added to the last sub-decoder, which drops out the shallow feature map after five decoding passes with a probability of P. Here, P is 0.05.

[0039] 4. The classifier part consists of a convolutional layer (Conv(N*3*3 / 1)) and an exponential normalization layer (Softmax). The size of the convolutional core in the convolutional layer is 3x3, the step length is 1, and the number is N, where N is the number of 2D semantic labels. In the exponential normalization layer (Softmax), dim=0, which means that the first dimension is normalized and the probability distribution of 2D semantic classification for each pixel is output.

[0040] Figure 3 shows a schematic diagram of the structure of the CSP-Res module in the encoder provided in this embodiment of the present invention. This is a cross-stage feature fusion layer (Partial Transition Layer), a hierarchical feature fusion mechanism that adopts the concept of pre-fusion and post-fusion, uses a gradient flow blocking strategy, and maximizes the difference between gradient combinations to prevent the residual network and other convolutional layers from learning overlapping gradient information, which theoretically improves the learning ability of the network. The specific steps are as follows: (11) 2C*H*W feature maps are input and passed through the left branch convolutional layer (c(C*1*1)), where the convolutional core size is 1×1, the step length is 1×1, and the number is C, to obtain C*H*W feature maps. (12) The 2C*H*W feature map is input and passed through the deep feature extraction layer in the right branch, which includes two convolutional layers (c(c*1*1)) and one stacked residual network (Res*N). The specific flow is as follows: (2a) 2C*H*W feature maps are input to the first convolutional layer c (c*1*1), where the size of the convolutional core is 1×1, the step length is 1×1, and the number of elements is C. (2b) The result of step (2a) is input into the second layer residual network structure (Res*N), which consists of N stacked residual blocks. The specific calculation flow of each residual block is as follows: (2b-1) The result of step (2a) is input to the first convolutional layer (c(C*1*1)) of the residual block, where the size of the convolutional core is 1×1, the step length is 1×1, and the number is C. (2b-2) The result of step (2b-1) is input to the second convolutional layer (c(C*3*3)) of the residual block, where the size of the convolutional core is 3×3, the step length is 1×1, and the number is C. (2b-3) Add the result of step (2b-2) to the result of step (2a). (2b-4) The result of step (2b-3) is input to the subsequent N-1 residual blocks for successive calculations. (13) The results of (2b-4) and (1) are concatenated on the first channel (Cat). (14) The result of step (13) is input to the last convolutional layer (c(2C*1*1)) for feature fusion, where the size of the convolutional core is 1×1, the step length is 1×1, and the number is 2C. The size of the final output feature map is 2C*H*W. 4 shows the overall flowchart of the data pre-processing part provided in the embodiment of the present invention, characterized in that the steps in the flowchart connect step 2 and step 3 in the specific embodiment. The specific operation steps include: (21) Analyze the lidar data to obtain a raw lidar point cloud frame, each of which has a size of N × 4, where N is the number of points in each frame, and 4 is the characteristic dimension of each point, i.e., [x, y, z, r], where (x, y, z) is the spatial position of the point cloud in the 3D Cartesian coordinate system, and r is the reflectance intensity of the surface material of the object where the point cloud is located. (22) The (x, y, z) in the original lidar point cloud frame is transformed into a polar coordinate system to obtain the spatial position representation (Ψ, θ, d) in the polar coordinate system, where Ψ is the horizontal deflection angle in the polar coordinate system, θ is the vertical deflection angle in the polar coordinate system, and d is the distance from the point to the origin of the coordinate system. The polar coordinate system transformation is as follows: TIFF2025542029000016.tif13140(4) TIFF2025542029000017.tif17140(5) TIFF2025542029000018.tif17140(6) The obtained (Ψ,θ,d) is combined with r to replace the original four-dimensional feature, and a new four-dimensional feature (Ψ,θ,d,r) is obtained. Ψ is the pitch angle in the polar coordinate system of the point cloud, θ is the deflection angle in the polar coordinate system of the point cloud, and d is the distance feature from the point cloud to the origin of the radar coordinate system. After feature transformation, a four-fold feature, i.e., (Ψ,θ,d,r), is constructed. r is the reflectivity of the surface material of the object where the point cloud is located.

[0041] (23) Using the spherical projection method, the spatial position features (x, y, z) of each point in the Cartesian coordinate system are used, combined with the parameters of the onboard lidar itself, to obtain the (h, w) index pair for each point, where h is the column index of the range image and w is the row index of the range image. The spherical projection is as shown in the following formula (7-9). f=f down +f up(7) TIFF2025542029000019.tif17140(8) TIFF2025542029000020.tif20133(9) where f is the range of the lidar vertical field of view, and f down is the lower limit of the lidar vertical field of view, f up is the lidar vertical field of view upper limit, H is the vertical resolution of the range feature map, typically the lidar beam, W is the horizontal resolution of the range feature map, TIFF2025542029000021.tif15133 is a downward-boundary function. Based on the size of feature d, the quadruple features and (h, w) of all point clouds in the single frame obtained in step 2 are sorted in descending order. The re-sorted quadruple features are mapped to corresponding 2D distance feature maps using (h, w). These feature maps are then stitched together and normalized to obtain a distance feature map with dimensions of 4×H×W. H is the vertical resolution of the distance feature map, and W is the horizontal resolution of the distance feature map. (24) A normalization process is applied to the distance feature map with a size of 4 × H × W to obtain the final feature map. The normalization process is as shown in formula (10) below. TIFF2025542029000022.tif19133(10) Among them, x i is the feature value on each feature channel of the feature map, μ i is the feature mean value on the corresponding feature channel, σ i is the feature variance on the corresponding feature channel, where i is from 0 to 3. This method performs normalization processing on each feature channel, and finally obtains a normalized 4-channel distance feature map.

[0042] (25) Using the (h, w) index pairs obtained in step (23), the original discrete 3D semantic labels are first converted to continuous labels using a label mapping table. Then, (h, w) is mapped to a 2D image, resulting in a single-channel 2D semantic image with dimensions H × W. To distinguish between major categories during the data labeling process, the correct label indices are discontinuous, and continuous correct labels are required for the subsequent semantic inference stage. At the same time, for some scenes, no distinct semantic classification scenes emerge. Learning the characteristics of such labels can easily interfere with the model. Therefore, it is necessary to prevent the model from overlearning and interfering with the classification information. Therefore, it is necessary to map such labels to existing labels to achieve label merging and continuation. The label merging process is as follows: label mapped ←index[label origin ] Among them, label mapped is the label after mapping, label origin is a primitive label, and index is a mapping relation dictionary. A specific example is shown in Table (1) below.

[0043] TIFF2025542029000023.tif157161

[0044] In this example, the three major classification labels with original label indexes 0 to 49 are mapped as consecutive indices 0 to 12. The input for this process is 1×N original point cloud labels (label origin where N is the number of points and n is the number of classifications. class and the mapping relationship dictionary has the following format: map_dict:key∈(0,max(origin label ),value∈(0,n class ) Among them, map_dict is a mapping dictionary, key is the key of the dictionary, value is the value of the dictionary, and origin label is the index value of the original label, and max() is the function that takes the maximum value. The final output result is a single-channel 2D semantic map label mapped is.

[0045] It is clear that the above-described examples of the present invention are merely examples for clearly explaining the operations of the present invention, and do not limit the embodiments of the present invention. Those skilled in the art can make various changes and modifications based on the above description, but it is not necessary to and cannot cover all embodiments here. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the claims of the present invention.

[0046] In the lightweight fully convolutional neural network training process described in step 4, the loss function adopted is the weighted cross-entropy loss combined model structure loss, and the warm-up learning rate mechanism in random gradient descent is used to dynamically adjust the learning rate to improve convergence efficiency and accuracy. The loss function is as shown in formula (11-14) below. TIFF2025542029000024.tif19133(11) TIFF2025542029000025.tif21133(12) TIFF2025542029000026.tif17133(13) loss=L1+L2(14) Among them, L1 is the weighted cross-entropy loss, C is the classification label, and w C is the weighting of the classification C, and N is the number of classification labels, which is set to 20 in this example as a specific example. TIFF2025542029000027.tif14133 is the probability estimate that the label after model inference for a training set sample is C, and y Cis the true probability that the sample label is C, and f C is the proportion of samples classified as C to all samples, ε is the offset amount, which is set to 0.001 in this example as a specific example. L2 is the structural loss of the model, and α is the proportional coefficient of the loss, which is set to 0.5 in this example as a specific example. w is a one-dimensional vector consisting of all parameters of the model, TIFF2025542029000028.tif13133 is the 2-norm and loss is the total loss of the model.

[0047] 5 shows a schematic configuration diagram of a point cloud semantic segmentation device provided in an embodiment of the present invention, which specifically includes: an acquisition module for acquiring the original point cloud data; an input module for preprocessing the source point cloud data and extracting preprocessed features of each point in the point cloud data by a first convolutional layer; an encoding module for performing feature extraction and downsampling on the input point cloud features using a predetermined number of encoding layers to obtain intermediate feature mappings corresponding to each of the encoding layers, the encoding layers including: a downsampling module, a cross-stage feature fusion residual module, and a dropout module; a decoding module for performing feature decoding and upsampling on the intermediate feature mapping using a predetermined number of decoding layers to obtain an upsampled feature mapping corresponding to each of the decoding layers, wherein the decoding layers include an upsampling module and a residual module; a classification module for collectively summarizing the intermediate feature mappings generated by the decoding layer and performing point-wise two-dimensional semantic classification using the holistic features; a mapping module for remapping the two-dimensional semantic labels generated by the classification module into three-dimensional space to generate three-dimensional semantic labels; An output module for aggregating the point-wise 3D semantic labels output by the mapping module for each scene frame and outputting a 3D semantic scene frame.

[0048] 6 shows a schematic diagram of the configuration of an electronic device and medium provided in an embodiment of the present invention. Specifically, the electronic device and medium include a lidar, a processor, a removable storage medium, a communication unit, an on-board computing cluster, a memory, and a computer program stored and executed in the memory. The lidar is a rotary lidar with a horizontal field of view range of 360°. When the processor executes the program, it realizes the deep learning-based 3D lidar point cloud semantic segmentation method according to any one of claims 1 to 9. The on-board computing cluster is a graphics processing unit (GPU) cluster that realizes the training and inference process according to any one of claims 6 to 9.

[0049] The external electronic devices and media include a solid, semi-solid, or mechanical rotating lidar connected to an input / output (I / O) interface, which can capture information in the environment to generate raw point cloud data and input it as a subsequent model; a communication unit including a network interface card such as a LAN card or a modem, which can perform communication processing via a network such as the Internet; and a removable storage medium including a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., into which a computer reads a computer program stored therein from the I / O port or writes a data frame of a semantic segmentation result that is not permanently stored, as required. The internal electronic devices and media described below include a central processing unit (CPU) that can execute various commands and operations by loading programs stored in read-only memory (ROM) into random access memory (RAM), where various programs and data necessary for the device's operation are also stored in the RAM. The on-board computing cluster receives commands from the CPU and data from the ROM to perform large-scale data parallel calculations. In particular, in this embodiment, it is an on-board graphics processing unit (GPU) cluster connected via a high-speed bus (PCI-E) inside the computer, and the CPU, on-board computing cluster, ROM, and RAM are connected to each other via the bus. The I / O interface is also connected to the bus. In particular, according to embodiments of the present disclosure, the processes described above with reference to Figure 1 may be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product based on deep learning and a lidar point cloud semantic segmentation method, tangibly including a computer program on a device-readable medium, the computer program including program code for executing the semantic segmentation method. In such embodiments, the computer program may be downloaded and installed over a network through a communication portion and / or installed from a removable storage medium.

[0050] The systems, devices, modules, or units described in the above embodiments can be specifically realized by computer chips or entities, or by products having certain functions. A typical realization device is an in-vehicle computer. Specifically, the in-vehicle computer can be, for example, a personal computer, a laptop computer, a tablet computer, a workstation, or any combination of these devices.

[0051] In another aspect, the present application also provides a storage medium, which may be included in the apparatus described above in the above embodiments, or may be a standalone storage medium not incorporated in the device. The storage medium stores one or more programs that are used by one or more processors to execute the deep learning-based 3D LIDAR point cloud semantic segmentation method described in the present invention. The storage medium may be persistent or non-persistent, removable or non-removable, and may be implemented by any method or technology. The information may be computer instructions, program modules, or other data, such as source data, result data, etc. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, or other optical or magnetic memories used to store information accessible by a computing device.

Claims

1. In a method for three-dimensional lidar point cloud semantic segmentation based on deep learning, Step 1 of constructing a lightweight deep fully convolutional neural network for training, the fully convolutional neural network consisting of three parts: an encoder, a decoder, and a classifier; Step 2 is a step of performing feature transformation on the original LIDAR point cloud, in which an original 3D point cloud dataset is obtained from the LIDAR, and reduced to four feature dimensions through feature transformation and joining; Step 3: converting the three-dimensional spatial features into two-dimensional spatial features and performing preprocessing, which involves generating a structured four-channel distance feature map from the chaotic features using a spherical projection method, generating a single-channel two-dimensional semantic image from the sequence labels, and converting it into a format that matches the format required for the preprocessed data, preprocessing the data, and dividing the preprocessed data samples into a training set and a test set; Step 4: training a lightweight deep fully convolutional neural network, in which the point cloud feature maps in the training set are used as training samples, the point cloud semantic map is used as ground truth labels, and the deep fully convolutional semantic segmentation neural network is trained using backpropagation and random gradient descent to minimize a loss function and obtain final optimized model parameters; Step 5: performing semantic segmentation on the 2D point cloud, inputting the source point cloud into a 2D semantic segmentation frame and obtaining 2D spatial semantic labels; Step 6: post-processing and optimizing the 2D semantic labels, using a sliding window voting mechanism based on a 2D matrix, and taking advantage of the consistency of point cloud density within a single object and the invariance of rotation and translation to comprehensively correct labeling errors caused by the small amount of single-class point cloud frame data, the sparseness of point cloud due to long-distance sampling, and the vagueness of semantic boundaries; and (7) a step of recovering 3D semantic labels, in which the label results after 2D semantic segmentation are de-mapped into a raw 3D point cloud semantic space to obtain a complete point cloud frame with 3D semantic labels. A method for 3D lidar point cloud semantic segmentation based on deep learning.

2. The lightweight deep fully convolutional network described in step 1 includes an input layer, a CSPDarknet encoder consisting of a cross-stage partial feature fusion strategy (CSP) and a residual structure, a decoder based on a deconvolution structure, and a classifier for pixel-by-pixel classification, constituting a point cloud segmentation network, abbreviated as CSP RangeNet, in which the input is a preprocessed multi-channel 2D point cloud range image, and the output is a 2D semantic label corresponding to each pixel of the range image; The input layer consists of a convolution layer, a batch normalization layer, and a Mish activation function layer (Conv-BN-M), The Mish activation function is as follows: Mish(x)=x×tanh(ξ(x))(1) （２） ξ(x)=log(1+e) ｘ )(3) Wherein, x is the input feature map, tanh(x) is the hyperbolic tangent function, ξ(x) is the exponential function with base e, the size of the convolution core in the convolution layer is 1 × 1, the number is 32, and the step length is 1 × 1. The input data undergoes feature transformation and preprocessing, resulting in a distance feature map of size C × H × W, where C is the number of feature channels (here 4), H is the number of vertical laser beams of the lidar, and W is the horizontal angular resolution of the distance feature map. The encoder is a CSP Darknet, which consists of a cross-stage feature fusion layer and a residual network, and is divided into five sub-encoders. Each encoder consists of a downsampling layer (DS), a cross-stage feature fusion residual layer (CSP-Res), and a dropout layer (D). Each downsampling layer consists of a convolutional layer - a batch normalization layer - a Mish activation function layer (C(3*3 / 2)-BN-M). The size of the convolutional core is 3x3, the step length is 1x2, and the numbers in the five sub-encoders are [64, 128, 256, 512, 1024], respectively. After passing through this layer, the distance feature map remains unchanged in the H dimension and in the W dimension. The feature extraction structure of each sub-encoder is composed of a cross-stage feature fusion residual layer (CSP-Res), which differs in the number of residual blocks. The numbers in the five sub-encoders are [1, 2, 8, 8, 4], respectively. The cross-stage feature fusion residual layers are [CSP-Res1, CSP-Res2, CSP-Res8, CSP-Res8, CSP-Res4], respectively. Deep features can be extracted from the downsampled feature map through this layer. The dropout layer (D) drops out the extracted deep feature map with a probability P. The CSP Darknet encoder includes five CSP-Res module structure diagrams. It is a cross-stage feature fusion layer (Partial Transition Layer), which is a hierarchical feature fusion mechanism. It adopts the concept of pre-fusion and post-fusion, and uses a gradient flow blocking strategy to maximize the difference between gradient combinations, thereby preventing the residual network and other convolutional layers from learning overlapping gradient information. This theoretically improves the learning ability of the network. Specifically, (11) Input 2C*H*W feature maps and pass them through the left branch convolution layer (c(C*1*1)), in which the size of the convolution core is 1x1, the step length is 1x1, and the number is C, to obtain C*H*W feature maps; (12) The 2C*H*W feature map is input and passed through the deep feature extraction layer of the right branch, which includes two convolution layers (c(c*1*1)) and one stacked residual network (Res*N). The specific flow is as follows: (2a) 2C*H*W feature maps are input to the first convolutional layer c (c*1*1), of which the size of the convolutional core is 1*1, the step length is 1*1, and the number is C; (2b) inputting the result of step (2a) into a second layer residual network structure (Res*N), where the layer is composed of N stacked residual blocks; The specific calculation flow for each residual block is as follows: (2b-2) The result of step (2a) is input into the first convolutional layer (c(C*1*1)) of the residual block, in which the size of the convolutional core is 1×1, the step length is 1×1, and the number is C; (2b-2) The result of step (2b-1) is input to the second convolutional layer (c(C*3*3)) of the residual block, in which the size of the convolutional core is 3*3, the step length is 1*1, and the number is C; (2b-3) adding the result of step (2b-2) to the result of step (2a); (2b-4) The result of step (2b-3) is input to the subsequent N-1 residual blocks for successive calculations; (13) concatenating (Cat) the result of step (2b-4) and the result of step (11) on the first channel; (14) The result of step (13) is input into the final convolution layer (c(2C*1*1)) to perform feature fusion, in which the size of the convolution core is 1×1, the step length is 1×1, and the number is 2C, and the size of the final output feature map is 2C*H*W; The decoder consists of five sub-decoders, each of which consists of an upsampling module (US) and a residual block (Res). The upsampling module has a deconvolution structure (ConvTransposed2d(4*4 / 2)), with a convolution core size of 4x4 and a step length of 2. The numbers in the five sub-decoders are [1024, 512, 256, 128, 64] in order. The features extracted by the backbone encoder are upsampled, i.e., increased by 32 times in W dimensions. Then, the original feature map is restored to the resolution H×W, and one residual block (Res) is added to improve the result. At the same time, after each upsampling, a jump connection is added between the outputs of the encoders with different resolutions, i.e., the decoder output is added to the corresponding encoder output to recover some of the high-frequency edge information lost in the downsampling process. In the last sub-decoder, a dropout layer (D) is added to drop out the shallow feature map after five decoding passes with a probability P, where P is 0.

05. The classifier unit is composed of a convolution layer (Conv(N*3*3 / 1)) and an exponential normalization layer (Softmax), in which the size of the convolution core in the convolution layer is 3x3, the step length is 1, and the number is N, where N is the number of two-dimensional semantic labels. In the exponential normalization layer (Softmax), dim=0, i.e., the first dimension is normalized and a probability distribution of two-dimensional semantic classification is output for each pixel point. The method for deep learning based 3D lidar point cloud semantic segmentation according to claim 1.

3. The source point cloud feature transformation in step 2 generates new features by using a coordinate system transformation method to generate four-dimensional features (Ψ, θ, d, r). (21) Analyzing the lidar data to obtain original lidar point cloud frames, each frame having a size of N×4, where N is the number of points in each frame, and 4 is the characteristic dimension of each point, i.e., [x, y, z, r], where (x, y, z) is the spatial position of the point cloud in a three-dimensional Cartesian coordinate system, and r is the reflectance intensity of the surface material of the object where the point cloud is located; (22) Convert (x, y, z) in the original LIDAR point cloud frame into a polar coordinate system to obtain a spatial position representation (Ψ, θ, d) in the polar coordinate system, where Ψ is the horizontal deflection angle in the polar coordinate system, θ is the vertical deflection angle in the polar coordinate system, and d is the distance from the point to the origin of the coordinate system. The polar coordinate system conversion is as follows: （４） （５） （６） combining the obtained (Ψ, θ, d) with r to replace the original four-dimensional feature to obtain a new four-dimensional feature (Ψ, θ, d, r); Wherein, Ψ is the pitch angle of the point cloud in the polar coordinate system, θ is the deflection angle of the point cloud in the polar coordinate system, and d is the distance feature from the point cloud to the origin of the radar coordinate system. After feature transformation, a quadruple feature, i.e., (Ψ, θ, d, r), is constructed, and r is the reflectivity of the surface material of the object where the point cloud is located. The method for deep learning based 3D lidar point cloud semantic segmentation according to claim 1.

4. Transforming the three-dimensional spatial features according to step 3 into two-dimensional spatial features to generate a structured four-channel distance feature map; (31) Using a spherical projection method, the spatial position characteristics (x, y, z) of each point in the Cartesian coordinate system are used, and combined with the parameters of the vehicle-mounted lidar itself to obtain an (h, w) index pair for each point, where h is the column index of the range image and w is the row index of the range image. The spherical projection is as shown in the following formula (7-9): f=f ｄｏｗｎ +f ｕｐ (7) （８） （９） Wherein, f is the range on the lidar vertical field of view, and f ｄｏｗｎ is the lower limit of the lidar vertical field of view, f ｕｐ is the lidar vertical field of view upper limit, H is the vertical resolution of the range feature map, typically the lidar beam, W is the horizontal resolution of the range feature map, is a downward-boundary function, and according to the size of the feature d, the quadruple features and (h, w) of all the point clouds of the single frame obtained in step 2 are arranged in descending order, and the rearranged quadruple features are mapped to corresponding two-dimensional distance feature maps by (h, w), and these feature maps are stitched together and normalized to obtain a distance feature map with a size of [4×H×W], where H is the vertical resolution of the distance feature map and W is the horizontal resolution of the distance feature map; (32) A normalization process is applied to a distance feature map of size 4×H×W to obtain a final feature map, and the normalization process is as follows: （１０） Among them, x ｉ is the feature value on each feature channel of the feature map, μ ｉ is the feature mean value on the corresponding feature channel, σ ｉ is the feature variance on the corresponding feature channel, and the method involves performing normalization on each feature channel to obtain a normalized four-channel distance feature map; (33) Using the (h, w) index pairs obtained in step (31), first generate continuous labels through label mapping using a label mapping table for the original discontinuous 3D semantic labels, and then map (h, w) to a 2D image to obtain a single-channel 2D semantic image with a size of H x W. In order to distinguish each major classification during the data labeling process, the correct label indexes are discontinuous, and continuous correct labels are required in the subsequent semantic inference stage. At the same time, for some scenes, individual semantic classification scenes do not appear, and if the model learns the features of such labels, they are likely to interfere with the model. Therefore, it is necessary to prevent the model from over-learning and interfering with the classification information. Therefore, it is necessary to map such labels with existing labels to realize label combination and continuation, and the label combination process is as follows: label ｍａｐｐｅｄ ←index[label ｏｒｉｇｉｎ ] Among them, label ｍａｐｐｅｄ is the label after mapping, label ｏｒｉｇｉｎ is the original label, index is the mapping relation dictionary, and the input in this process is 1×N original point group label ｏｒｉｇｉｎ where N is the number of points and n is the number of classifications. ｃｌａｓｓ And the format of the mapping relationship dictionary is: map_dict:key∈(0, max(origin ｌａｂｅｌ ), value∈(0, n ｃｌａｓｓ ) Among them, map_dict is a mapping dictionary, key is a dictionary key, value is a dictionary value, origin ｌａｂｅｌ is the index value of the original label, max() is the function that takes the maximum value, and the final output result is the single-channel two-dimensional semantic map label after mapping. ｍａｐｐｅｄ characterized in that The method for deep learning based 3D lidar point cloud semantic segmentation according to claim 1.

5. In the lightweight fully convolutional neural network training process described in step 4, the loss function adopted is weighted cross-entropy loss combined model structure loss, and a warm-up learning rate mechanism is adopted in random gradient descent to dynamically adjust the learning rate and improve the convergence efficiency and convergence accuracy. The loss function is as shown in the following formula (11-14): （１１） （１２） （１３） loss=L1+L2(14) Among them, L1 is the weighted cross-entropy loss, C is the classification label, and w Ｃ is the weight of classification C, N is the number of samples, is the probability estimate of C for the label after model inference for the training set sample, y Ｃ is the true probability that the sample label is C, and f Ｃ is the proportion of samples classified as C to all samples, ε is the offset amount, L2 is the structural loss of the model, α is the proportional coefficient of the loss, w is a one-dimensional vector consisting of all parameters of the model, is the 2-norm and loss is the total loss of the model. The method for deep learning based 3D lidar point cloud semantic segmentation according to claim 1.

6. The method for 3D LIDAR point cloud semantic segmentation based on deep learning as described in claim 1, characterized in that in the 2D point cloud semantic segmentation process described in step 5, the original point cloud undergoes preprocessing in steps 2, 3 and 4, and then is input into a lightweight deep fully convolutional neural network to perform point-by-point inference, thereby obtaining 2D spatial semantic labels.

7. The method for 3D LIDAR point cloud semantic segmentation based on deep learning as claimed in claim 1, characterized in that the sliding window voting mechanism based on the 2D matrix in step 6 determines the pixel points within the region range based on a distance threshold for each pixel point in the 2D semantic label map obtained in step 5, and votes based on the label, thereby performing a comprehensive modification of the 2D semantic label. Since the operations of each point in the 2D matrix are all independent, the operation process can be accelerated by using a graphics processing unit (GPU) for parallel computing.

8. The method for 3D LIDAR point cloud semantic segmentation based on deep learning according to claim 1, characterized in that in the recovery of 3D semantic labels in step 7, a label demapping table is used to demap the optimized 2D semantic labels obtained in step 6 into 3D semantic space, and finally a complete point cloud frame with 3D semantic labels is obtained.

9. In a deep learning-based 3D lidar point cloud semantic segmentation device, an acquisition module for acquiring raw point cloud data; an input module for preprocessing the source point cloud data and extracting preprocessed features of each point in the point cloud data by a first convolutional layer; an encoding module for performing feature extraction and downsampling on the input point cloud features using a predetermined number of encoding layers to obtain intermediate feature mappings corresponding to each of the encoding layers, the encoding layers including: a downsampling module, a cross-stage feature fusion residual module, and a dropout module; a decoding module for performing feature decoding and upsampling on the intermediate feature mapping using a predetermined number of decoding layers to obtain an upsampling feature mapping corresponding to each of the decoding layers, wherein the decoding layer includes an upsampling module and a residual module; a classification module for collectively summarizing the intermediate feature mappings generated by the decoding layer and performing point-wise two-dimensional semantic classification using the holistic features; a mapping module for remapping the two-dimensional semantic labels generated by the classification module into a three-dimensional space to generate three-dimensional semantic labels; an output module for collating the three-dimensional semantic labels for each point output by the mapping module for each frame and outputting a three-dimensional semantic frame, A deep learning based 3D lidar point cloud semantic segmentation device.

10. An electronic device and medium includes a lidar, a processor, a removable storage medium, a communication unit, an on-board computing cluster, a memory, and a computer program stored in and executed by the memory, wherein the lidar is a rotary lidar with a horizontal field of view range of 360°, and the processor, when executing the program, realizes the deep learning-based 3D lidar point cloud semantic segmentation method according to any one of claims 1 to 4, and the on-board computing cluster is a graphics processing unit (GPU) cluster that realizes the training and inference process according to any one of claims 2 to 4; The external electronic devices and media include a solid, semi-solid, or mechanical rotating lidar connected to an input / output (I / O) interface, which can capture information in the environment to generate raw point cloud data and input it as a subsequent model; a communication section including a network interface card such as a LAN card or a modem, which can perform communication processing via a network such as the Internet; and a removable storage medium including a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., into which a computer reads a computer program stored therein from the I / O port or writes a data frame of a semantic segmentation result that is not permanently stored, as required. The internal electronic devices and media described below include a central processing unit (CPU) that can execute various commands and operations by loading programs stored in read-only memory (ROM) into random access memory (RAM), and the RAM also stores various programs and data necessary for operating the device. The in-vehicle computing cluster receives commands from the CPU and data from the ROM and performs large-scale data parallel calculations. In particular, in this embodiment, the in-vehicle computing cluster is an in-vehicle graphics processing unit (GPU) connected via a high-speed bus (PCI-E) inside the computer, and the CPU, in-vehicle computing cluster, ROM, and RAM are connected to each other via the bus, and the I / O interface is also connected to the bus. Electronic devices and media.

11. The storage medium may be a storage medium included in the apparatus of claim 10, or may be a standalone storage medium not incorporated into a device, the storage medium having one or more programs stored therein, the programs being used by one or more processors to execute the deep learning based 3D LIDAR point cloud semantic segmentation method described in the present invention, the storage medium may include persistent and non-persistent, removable and non-removable media, and may achieve storage of information by any method or technology, the information being computer instructions, program modules, or other data such as source data, result data, etc. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technologies, or other optical memory, magnetic memory used to store information accessible by a computing device.

Images