A laser radar point cloud segmentation method, system, device and storage medium

Abstract

Embodiments of the present application relate to the technical field of three-dimensional scene completion, in particular to a laser radar point cloud segmentation method, system, device and storage medium, the method comprising: taking a rasterized distance view image as an input of a segmentation network, mapping the rasterized distance view image to a high-dimensional embedding to obtain a feature map; performing attention calculation on the feature map to obtain features corresponding to the feature map; performing semantic category prediction on the features to obtain probability distribution of each grid position in the distance view image for different semantic categories; taking the probability distribution as an output of the segmentation network, and projecting the network output to a three-dimensional space to complete semantic segmentation of the laser radar point cloud. The laser radar point cloud segmentation method provided by the present application can model long-distance range features under a global receptive field to better solve the problems existing in laser radar point cloud segmentation based on distance view representation.

Description

Technical Field

[0001] This application relates to the fields of deep learning and autonomous driving perception technology, and in particular to a lidar point cloud segmentation method, system, device and storage medium. Background Technology

[0002] LiDAR point cloud segmentation is a crucial task in autonomous driving perception. In recent years, various LiDAR point cloud segmentation algorithms based on different representations have been proposed. Among them, range view-based representation is a common and practical approach. This representation projects points in 3D space onto a compactly arranged 2D range view image, achieving a 3D-to-2D rasterization. This transforms the 3D LiDAR point cloud segmentation problem into a 2D range view image segmentation problem, thereby improving the compactness of the point cloud representation and reducing computational complexity.

[0003] Existing LiDAR point cloud segmentation algorithms based on range view representations, namely RangeNet++, FIDNet, and CENet, are all composed of fully convolutional networks. Because convolutional operations are inherently limited by a small receptive field, they struggle to model feature representations over long distances. LiDAR point cloud segmentation based on range view representations rasterizes points in 3D space to 2D space; this transformation process is constrained by factors such as the resolution of the range view image, leading to a series of potential transformation problems. However, segmentation models based on fully convolutional networks, lacking global receptive fields and the ability to model features over long distances, are unable to alleviate these problems. Summary of the Invention

[0004] This application provides a lidar point cloud segmentation method, system, device, and storage medium that can model long-range features under a global receptive field to better solve the problems existing in lidar point cloud segmentation based on distance view representation.

[0005] To address the aforementioned technical problems, in a first aspect, embodiments of this application provide a LiDAR point cloud segmentation method, comprising the following steps: using a rasterized range view image as input to a segmentation network, mapping the rasterized range view image to a high-dimensional embedding to obtain a feature map; performing attention calculation on the feature map to obtain features corresponding to the feature map; performing semantic category prediction on the features to obtain the probability distribution of each grid position in the range view image for different semantic categories; using the probability distribution as the output of the segmentation network, and projecting the network output onto a three-dimensional space to complete the semantic segmentation of the LiDAR point cloud.

[0006] In some exemplary embodiments, before sending the rasterized distance view image to the segmentation network, the method includes: performing data augmentation processing on the rasterized distance view image.

[0007] In some exemplary embodiments, before sending the rasterized distance view image to the segmentation network, the method further includes: cutting the rasterized distance view image to obtain a plurality of sub-distance view images; and using the sub-distance view images as input to the segmentation network.

[0008] In some exemplary embodiments, the rasterized range view image is obtained by rasterizing the lidar point cloud. The rasterization process includes: projecting each point in the lidar point cloud onto a grid of the range view image to obtain a point cloud input; the point cloud input is characterized by the two-dimensional range view image; and the point cloud input includes the following features: point coordinates, depth, reflection intensity, and mask; wherein the mask is used to characterize whether each grid position in the point cloud input contains a valid point.

[0009] In some exemplary embodiments, attention is calculated on the feature map using formula (1), which is shown below: (1) in, This is the output of the multi-head attention module; This is a self-attention mechanism; The formula for calculating attention; Represents the normalized exponential function, W represents the dimensions of the head. Q W K W V W O These are the weight matrices for the query Q, key K, value V, and output O in the self-attention mechanism, respectively.

[0010] Secondly, embodiments of this application also provide a lidar point cloud segmentation system, comprising: a segmentation network and a projection module connected to the segmentation network, wherein the input of the segmentation network is a rasterized distance view image, and the output of the segmentation network is the probability distribution of each grid position in the distance view image for different semantic categories; the segmentation network is used to map the rasterized distance view image into a high-dimensional embedding to obtain a feature map; attention is calculated on the feature map to obtain features corresponding to the feature map; and semantic category prediction is performed on the features to obtain the probability distribution of each grid position in the distance view image for different semantic categories; the projection module is used to project the network output onto a three-dimensional space to complete the semantic segmentation of the lidar point cloud.

[0011] In some exemplary embodiments, the segmentation network includes a distance embedding module, a segmentation module, and a semantic segmentation head module connected in sequence; both the distance embedding module and the semantic segmentation head module include several layers of multilayer perceptrons; the segmentation module includes multiple sub-segmentation modules connected in sequence, each sub-segmentation module including a block embedding module, an addition and normalization module, a forward module, a multi-head attention module, and a block aggregation module; the distance embedding module is used to map the distance view image into a high-dimensional embedding to obtain a feature map; the segmentation module is used to feed the feature map into the sub-segmentation module for attention calculation to perform feature extraction and obtain features for each stage; the stages correspond one-to-one with the sub-segmentation modules; the semantic segmentation head module is used to perform semantic category prediction using the features of each stage.

[0012] In some exemplary embodiments, the forward module includes several layers of multilayer perceptrons and activation functions; the forward module is characterized by formula (2): (2) in, Indicates the forward module; Indicates a linear layer; This indicates a residual connection.

[0013] In addition, this application also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described lidar point cloud segmentation method.

[0014] In addition, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described lidar point cloud segmentation method.

[0015] The technical solution provided in this application has at least the following advantages: This application provides a method, system, device, and storage medium for LiDAR point cloud segmentation. The method includes the following steps: using a rasterized range view image as input to a segmentation network, mapping the rasterized range view image to a high-dimensional embedding to obtain a feature map; performing attention calculation on the feature map to obtain features corresponding to the feature map; performing semantic category prediction on the features to obtain the probability distribution of each grid position in the range view image for different semantic categories; using the probability distribution as the output of the segmentation network, and projecting the network output onto a three-dimensional space to complete the semantic segmentation of the LiDAR point cloud.

[0016] The LiDAR point cloud segmentation method provided in this application is a self-attention mechanism-based algorithm. This algorithm uses a range view representation, aiming for superior segmentation performance compared to existing algorithms while maintaining high inference speed, thus achieving a better trade-off between performance and speed. Unlike previous segmentation algorithms constructed from fully convolutional networks, the LiDAR point cloud segmentation method proposed in this application is built using a self-attention mechanism. This mechanism can model long-range features within the global receptive field, better addressing the problems existing in LiDAR point cloud segmentation based on range view representations. Attached Figure Description

[0017] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments, and unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0018] Figure 1 This is a schematic diagram of a lidar point cloud segmentation process based on distance view representation.

[0019] Figure 2 This is a flowchart illustrating a lidar point cloud segmentation method provided in one embodiment of this application.

[0020] Figure 3 A framework diagram of SCPNet provided for an embodiment of this application.

[0021] Figure 4 This is a schematic diagram of the structure of a lidar point cloud segmentation system provided in an embodiment of this application.

[0022] Figure 5 This is a schematic diagram of the structure of a segmentation network provided in an embodiment of this application.

[0023] Figure 6 This is a schematic diagram of the structure of a segmentation network provided in another embodiment of this application.

[0024] Figure 7 This is a schematic diagram of the structure of a sub-segmentation module provided in an embodiment of this application.

[0025] Figure 8 The performance of the lidar point cloud segmentation method provided in one embodiment of this application on the SemanticKITTI dataset is shown in the figure, and the data comparison is shown with previous methods.

[0026] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] As the background technology shows, existing LiDAR point cloud segmentation algorithms are limited by the small receptive field of convolution operations, making it difficult to model feature representations over long distances. LiDAR point cloud segmentation based on distance view representation rasterizes points in three-dimensional space to two-dimensional space. This transformation process is constrained by factors such as the resolution of the distance view image, leading to a series of potential transformation problems.

[0028] LiDAR point cloud segmentation is a crucial task in autonomous driving perception. In recent years, various LiDAR point cloud segmentation algorithms based on different representations have been proposed, including point-based, voxel-based, distance-view-based, bird's-eye view-based, and multi-representation fusion-based algorithms. Among these methods, point-, voxel-based, and multi-representation fusion-based algorithms are often limited by neighborhood search, 3D operators, or multi-branch network structures, making it difficult to achieve fast or even real-time inference speeds. Thanks to their compact arrangement and efficient 2D operators (2DOperators), distance-view and bird's-eye view-based algorithms often exhibit faster inference speeds. However, the segmentation performance of existing distance-view and bird's-eye view-based algorithms significantly lags behind that of voxel-based and multi-representation fusion-based algorithms.

[0029] Currently, distance-view-based representation is a common and practical approach in LiDAR point cloud segmentation algorithms. This representation achieves 3D-to-2D rasterization by projecting points in 3D space onto a compactly arranged 2D distance-view image. Specifically, a rotating LiDAR mounted on the roof of an autonomous vehicle emits laser beams at a preset angle. The laser beams return after contacting an object, and their spatial position can be calculated from the time difference between emission and return. The LiDAR point cloud collected from the laser beams consists of N points in 3D space, each point recording 3D coordinate information p. x p y p z With laser beam reflection intensity p i The distance view is represented by a distance view image with a width of W and a height of H, which arranges points in three-dimensional space according to three-dimensional coordinate information and projects them onto the distance view image.

[0030] A complete lidar point cloud segmentation process based on distance view representation is as follows: Figure 1 As shown, the lidar point cloud segmentation task based on range view representation uses the range view image as network input. A neural network model extracts features from the input data to predict the semantic category of each grid cell in the range view image. The predicted grid cells in the range view image are then projected back from two dimensions to three dimensions, thus achieving the segmentation of the lidar point cloud.

[0031] Existing LiDAR point cloud segmentation algorithms based on range view representations consist of fully convolutional networks, with representative schemes including RangeNet++, FIDNet, and CENet. RangeNet++ proposes a LiDAR point cloud segmentation network structure composed of fully convolutional networks. This structure takes a 3D-to-2D rasterized range view image as input, extracts features through convolutional operations, performs deconvolution on the features, makes predictions, and reprojects the raster in the predicted range view image back into 3D space to achieve semantic category segmentation of the LiDAR point cloud. FIDNet improves upon RangeNet++. It uses the standard ResNet residual network as the backbone of the LiDAR point cloud segmentation network and replaces the deconvolution operation with bilinear interpolation, thus improving the accuracy of point cloud segmentation. CENet further improves the network structure in FIDNet by using convolutional operations with a larger receptive field to aggregate information over a wider range during feature extraction.

[0032] Existing LiDAR point cloud segmentation algorithms based on range view representations, namely RangeNet++, FIDNet, and CENet, are all composed of fully convolutional networks. Because convolutional operations are inherently limited by a small receptive field, they struggle to model feature representations over long distances. LiDAR point cloud segmentation based on range view representations rasterizes points in 3D space to 2D space; this transformation process is constrained by factors such as the resolution of the range view image, leading to a series of potential transformation problems. However, segmentation models based on fully convolutional networks, lacking global receptive fields and the ability to model features over long distances, are unable to alleviate these problems.

[0033] To address the aforementioned technical problems, this application provides a LiDAR point cloud segmentation method, comprising the following steps: using a rasterized range view image as input to a segmentation network, mapping the rasterized range view image to a high-dimensional embedding to obtain a feature map; performing attention calculation on the feature map to obtain features corresponding to the feature map; performing semantic category prediction on the features to obtain the probability distribution of each grid position in the range view image for different semantic categories; using the probability distribution as the output of the segmentation network, and projecting the network output onto a three-dimensional space to complete the semantic segmentation of the LiDAR point cloud. The LiDAR point cloud segmentation method provided in this application can model long-range features within a global receptive field, thereby better addressing the problems existing in LiDAR point cloud segmentation based on range view representation.

[0034] The embodiments of this application will now be described in detail with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the embodiments of this application to facilitate a better understanding of the application. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.

[0035] See Figure 2 This application provides a lidar point cloud segmentation method, including the following steps: Step S1: Using the rasterized distance view image as input to the segmentation network, the rasterized distance view image is mapped into a high-dimensional embedding to obtain a feature map.

[0036] Step S2: Perform attention calculation on the feature map to obtain the features corresponding to the feature map; perform semantic category prediction on the features to obtain the probability distribution of each grid position in the distance view image for different semantic categories.

[0037] Step S3: Use the probability distribution as the output of the segmentation network and project the network output onto three-dimensional space to complete the semantic segmentation of the lidar point cloud.

[0038] The LiDAR point cloud segmentation method provided in this application is a self-attention mechanism-based algorithm. This algorithm uses a range view representation and aims to achieve better segmentation performance than existing algorithms while maintaining high inference speed, thus achieving a better trade-off between performance and speed. This application can model long-range features within the global receptive field to better address the problems existing in range view-based LiDAR point cloud segmentation. Simultaneously, to enable the model to learn more general representations during training and reduce memory consumption, this application also proposes a data augmentation method suitable for range view images and a view-segmentation method for LiDAR point cloud segmentation. These data augmentation and LiDAR point cloud segmentation methods achieve superior point cloud segmentation performance and lower training memory overhead on large-scale LiDAR point cloud segmentation datasets.

[0039] In some embodiments, before sending the rasterized range view image to the segmentation network, the LiDAR point cloud segmentation method provided in this application includes: performing data augmentation processing on the rasterized range view image. This application proposes a novel data augmentation method applicable to LiDAR point cloud segmentation algorithms based on range view representations. It uses the rasterized range view image as training data, generating new training data while using existing data and annotations, thereby enhancing the diversity of the training data and ultimately effectively improving the accuracy of LiDAR point cloud segmentation.

[0040] The following provides a detailed description of the data augmentation method provided in this application, which can be applied to lidar point cloud segmentation algorithms based on distance view representation.

[0041] Data augmentation can often increase the diversity of training data, thereby helping segmentation networks learn more general and effective representations. Previous data augmentation methods based on LiDAR point clouds mainly focus on the point level, such as global rotation, jitter, flipping, and random removal. This application proposes a novel data augmentation method applicable to LiDAR point cloud segmentation algorithms based on range view representations. This method consists of four data augmentation techniques: mixing two frames of point clouds rasterized into range view images (RangeMix), unifying two frames of point clouds rasterized into range view images (RangeUnion), pasting two frames of point clouds rasterized into range view images (RangePaste), and sliding a point cloud on a single frame rasterized into a range view image (RangeShift). This method generates new training data using existing data and annotations to enhance the diversity of training data and ultimately effectively improve the accuracy of LiDAR point cloud segmentation.

[0042] Specifically, the RangeMix operation blends two point clouds that have been rasterized into range view images. This blending process begins by merging the points according to their tilt angles. Grouping, where p x p y p z Here are the coordinates of the points in the aforementioned point cloud in three-dimensional space. Subsequently, the points in the two frames are interweaved at corresponding positions to form two new mixed point clouds, thereby enhancing the training data.

[0043] Specifically, the RangeUnion operation combines the point clouds of two frames rasterized into range view images. As mentioned earlier, due to the spatial sparsity of point clouds, most of the range view image raster does not contain valid points; these locations are masked by p. e Characterization. This application uses p e The empty grid cells in a point cloud frame are indexed, and valid points from another point cloud frame are inserted at the corresponding positions to enhance the training data.

[0044] Specifically, the RangePaste operation pastes two point clouds rasterized into distance view images. This application uses semantic annotations of the point clouds to index semantic categories with a small number of occurrences in the dataset. Specifically, given a point cloud frame, the points containing semantic categories with a small number of occurrences can be obtained through semantic annotation; this application pastes these points into another randomly sampled point cloud frame to augment the training data.

[0045] Specifically, the RangeShift operation is performed on a point cloud that has been rasterized into a range view image in a single frame. This operation slides the point cloud along the azimuth direction, where the azimuth is represented as... .

[0046] In some embodiments, before sending the rasterized range view image to the segmentation network, the lidar point cloud segmentation method provided in this application further includes: cutting the rasterized range view image to obtain several sub-range view images; and using the sub-range view images as input to the segmentation network. That is, the segmented sub-range view images (sub-point clouds) are used as input to the segmentation network (model), and after being predicted separately, they are reassembled into a complete point cloud.

[0047] The following is a detailed description of the lidar point cloud segmentation method suitable for segmented view images of distance view provided in this application.

[0048] Existing LiDAR point cloud segmentation algorithms are trained using the entire point cloud or range view image. Limited by the memory constraints of computing devices, previous algorithms often limited the rasterization resolution of the point cloud, such as using a smaller width W during LiDAR point cloud rasterization. While this saves memory, it introduces problems such as many-to-one conflicts, deformation, and semantic incoherence, thus affecting the performance of LiDAR point cloud segmentation.

[0049] This application proposes a novel viewpoint-based LiDAR point cloud segmentation training method (STR for short). This method is as follows: Figure 3 As shown, STR segments a point cloud or range view image into equally spaced sub-point clouds or sub-range view images and uses them for training, significantly reducing memory usage during model training. Specifically, this application divides a complete frame of point cloud rasterized into a range view image with width W and height H into several sub-point clouds at equal intervals according to the vehicle's perspective; wherein, the width of each sub-point cloud is W. train The height is H. This operation is performed before model training; the segmented sub-point clouds serve as input to the model and are reassembled into a complete point cloud after being predicted separately.

[0050] Sub-point cloud based on the azimuth of the point The method divides a complete point cloud frame, covering a 360-degree view, into Z non-overlapping sub-point clouds, i.e.: This operation ensures the feasibility of using high resolution (i.e., a large width W value) during the rasterization of point clouds, thereby alleviating problems such as many-to-one conflicts, deformation, and semantic incoherence caused by insufficient resolution.

[0051] During training, for each frame of point cloud, this application randomly selects one of its Z sub-point clouds for training. During inference, this application predicts all sub-point clouds in each frame of point cloud, and after prediction, performs the same operation as segmentation to concatenate them, thereby obtaining a prediction of a complete frame of point cloud. The STR training method significantly reduces the memory usage of the LiDAR point cloud segmentation model during training; that is, when training in this way, the memory overhead can be reduced to [amount missing]. Therefore, this application proposes a novel LiDAR point cloud segmentation method based on segmented viewpoints. Unlike previous methods that used the entire point cloud or range view image for training, the proposed segmented viewpoint method divides the point cloud or range view image into equally spaced sub-point clouds or sub-range view images and uses them for training, which greatly reduces the memory usage during model training.

[0052] In some embodiments, the rasterized distance view image is obtained by rasterizing the lidar point cloud, and the rasterization process includes: Each point in the LiDAR point cloud is projected onto a grid of the range view image to obtain the point cloud input.

[0053] The point cloud input is represented by a two-dimensional distance view image; and the point cloud input includes the following features: the coordinates of the points (p x p y p z ),depth Reflection intensity p i and mask p e ; where the mask p e Used to characterize whether each grid position in the point cloud input contains a valid point.

[0054] Specifically, the lidar point cloud collected by laser rays consists of N points existing in three-dimensional space, and each point records three-dimensional coordinate information p. x p y p z With laser beam reflection intensity p i Specifically, for a point p in a point cloud... n In this application, the image is projected onto a raster of a distance view image with width W and height H. This rasterization process can be described by the following formula:

[0055] in, For point p n Coordinates in the distance view image; For point p n The depth between the lidar sensor and the sensor; For the lidar sensor in the vertical direction (i.e. - The range of viewpoints in the direction, where and These represent the tilt angles (inclination) of the sensor in the upward and downward directions, respectively. Generally, the height H of the range view image is set by the parameters configured for the LiDAR sensor itself; for example, the height of a point cloud collected by a 64-line sensor is typically set to H=64. The width W of the range view image can be set according to specific needs.

[0056] By processing each point in the lidar point cloud according to the above formula, a point cloud input represented by a two-dimensional distance view image can be obtained. It consists of features from six channels, including: the coordinates of a point (p x p y p z ),depth Reflection intensity p i and mask p e The mask p e Used for characterization Check whether each grid position contains a valid point.

[0057] See Figure 4 This application also provides a lidar point cloud segmentation system, including: a segmentation network 101 and a projection module 102 connected to the segmentation network 101. The input of the segmentation network 101 is a rasterized distance view image, and the output of the segmentation network 101 is the probability distribution of each grid position in the distance view image for different semantic categories. The segmentation network 101 is used to map the rasterized distance view image into a high-dimensional embedding to obtain a feature map; attention is calculated on the feature map to obtain features corresponding to the feature map; and semantic category prediction is performed on the features to obtain the probability distribution of each grid position in the distance view image for different semantic categories. The projection module 102 is used to project the network output onto a three-dimensional space to complete the semantic segmentation of the lidar point cloud.

[0058] See Figure 4 In some embodiments, the segmentation network 101 includes a distance embedding module 1011, a segmentation module 1012, and a semantic segmentation head module 1013 connected in sequence; wherein, as Figure 5As shown, the distance embedding module 1011 and the semantic segmentation head module 1013 both include several layers of multilayer perceptrons; the segmentation module 1012 includes multiple sub-segmentation modules connected in sequence. Figure 6 A schematic diagram of the overall structure of the segmentation network is shown. (See diagram below.) Figure 7 As shown, each sub-segmentation module includes a block embedding module 121, an addition and normalization module 122, a forward module 123, a multi-head attention module 124, and a block aggregation module 125; the distance embedding module 1011 is used to map the distance view image into a high-dimensional embedding to obtain a feature map; the segmentation module 1012 is used to send the feature map into the sub-segmentation module for attention calculation to extract features and obtain features for each stage; wherein, the stage corresponds one-to-one with the sub-segmentation module, for example, the four sub-segmentation modules correspond to the four stages respectively; the semantic segmentation head module 1013 is used to perform semantic category prediction using the features of each stage.

[0059] In some embodiments, the forward module 123 includes several layers of multilayer perceptrons and activation functions; the forward module 123 can be characterized by the following formula:

[0060] in, Indicates the forward module; Indicates a linear layer; This indicates a residual connection.

[0061] It should be noted that the forward module 123 can be composed of a three-layer multilayer perceptron (MLP) and activation functions.

[0062] To achieve global receptive field and long-range feature modeling, this application proposes a novel LiDAR point cloud segmentation algorithm based on a self-attention mechanism. The network structure (segmentation network) of this algorithm is shown below. Figure 6 As shown. Specifically, given a batch of rasterized distance view images with width W and height H. As network input, it undergoes feature extraction via a Range Embedding Module (REM) and a segmentation backbone consisting of four stages, followed by semantic prediction by four segmentation heads. Each stage consists of, for example,... Figure 7The diagram shows the Transformer Block, which consists of several sub-segmentation modules, including a patch embedding module, a multi-head attention module, an add & normalization module, a feed forward module, and a patch merge module.

[0063] like Figure 5 As shown, the distance embedding module 1011 can be composed of three multi-layer perceptrons (MLPs). The distance embedding module will embed the distance view image... Mapped into a high-dimensional embedding to obtain The feature map. Then... Attention is calculated in the first sub-segmentation module (segmentation backbone stage). Specifically, the feature map has a width of W and a height of H. The data is divided into 3×3 patches, which serve as input to the sub-segmentation modules (Transformer Blocks). Each sub-segmentation module in each stage uses the same patch partitioning method and is fed into its respective module for feature extraction to obtain the features for each stage. Their spatial dimensions are as follows: , , and .

[0064] The multi-head attention module 124, as the main computation module in the sub-segmentation module, performs attention calculation on the feature map using the following formula in some embodiments:

[0065] in, This is the output of the multi-head attention module; This is a self-attention mechanism; The formula for calculating attention; Represents the normalized exponential function, W represents the dimensions of the head. Q W K W V W O These are the weight matrices for the query (Q), key (K), value (V), and output (O) in the self-attention mechanism, respectively.

[0066] The semantic segmentation head module 1013 utilizes features generated by the segmentation backbone composed of sub-segmentation modules at each stage. Semantic category prediction is performed. To avoid overly complex computations, this application uses a simple multilayer perceptron (MLP) as the segmentation head. Figure 5 As shown, the semantic segmentation head module 1013 can be composed of four interconnected multilayer perceptrons. First, the spatial dimensions output from the four stages are respectively... , , and The feature maps are then processed through a linear layer to unify the number of channels. Finally, the channel-unified feature maps are uniformly scaled to the same spatial size, H×W, via bilinear interpolation. For the i-th stage (i=1,2,3,4), the above process can be characterized by the following formula:

[0067] in, Indicates bilinear interpolation; This represents a linear layer, i.e., a multilayer perceptron. Finally, the four feature maps with unified channel number and spatial size are... They are concatenated together according to the channel dimension, and the number of channels is mapped to each other through two linear layers. This refers to the number of semantic categories in the dataset, which forms the probability distribution of each grid position in the distance view image for different semantic categories. This probability distribution is the output of the self-attention mechanism-based LiDAR point cloud segmentation network proposed in this patent, which can be re-rasterized into three-dimensional space to complete the semantic segmentation of the LiDAR point cloud.

[0068] Compared with existing technologies, on the one hand, the self-attention mechanism-based lidar point cloud segmentation algorithm proposed in this application achieves better point cloud segmentation performance; on the other hand, the self-attention mechanism-based lidar point cloud segmentation algorithm proposed in this patent greatly reduces the memory usage and overhead required during training.

[0069] The feasibility of the self-attention mechanism-based LiDAR point cloud segmentation algorithm proposed in this application was demonstrated through detailed experiments and analyses on three standard LiDAR point cloud segmentation datasets. Specifically, this application used SemanticKITTI, a large dataset that collects point cloud data using a 64-line LiDAR, and performed performance verification of the proposed algorithm using a standard dataset partitioning method, namely, corresponding training, validation, and testing partitions. The experimental results are shown in the attached figures. Figure 8 As shown.

[0070] refer to Figure 9Another embodiment of this application provides an electronic device, including: at least one processor 110; and a memory 111 communicatively connected to the at least one processor; wherein the memory 111 stores instructions executable by the at least one processor 110, the instructions being executed by the at least one processor 110 to enable the at least one processor 110 to perform any of the above method embodiments.

[0071] The memory 111 and processor 110 are connected via a bus, which may include any number of interconnecting buses and bridges, connecting various circuits of one or more processors 110 and memory 111. The bus may also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver may be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 110 is transmitted over a wireless medium via an antenna, which further receives data and transmits it to processor 110.

[0072] Processor 110 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 111 can be used to store data used by processor 110 during operation.

[0073] Another embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0074] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0075] Based on the above technical solutions, this application provides a method, system, device, and storage medium for LiDAR point cloud segmentation. The method includes the following steps: using a rasterized range view image as input to a segmentation network, mapping the rasterized range view image to a high-dimensional embedding to obtain a feature map; performing attention calculation on the feature map to obtain features corresponding to the feature map; performing semantic category prediction on the features to obtain the probability distribution of each grid position in the range view image for different semantic categories; using the probability distribution as the output of the segmentation network, and projecting the network output onto a three-dimensional space to complete the semantic segmentation of the LiDAR point cloud.

[0076] The LiDAR point cloud segmentation method provided in this application is a self-attention mechanism-based algorithm. This algorithm uses a range view representation, aiming for superior segmentation performance compared to existing algorithms while maintaining high inference speed, thus achieving a better trade-off between performance and speed. Unlike previous segmentation algorithms constructed from fully convolutional networks, the LiDAR point cloud segmentation method proposed in this application is built using a self-attention mechanism. This mechanism can model long-range features within the global receptive field, better addressing the problems existing in LiDAR point cloud segmentation based on range view representations.

[0077] Those skilled in the art will understand that the above-described embodiments are specific examples of implementing this application, and in practical applications, various changes in form and detail may be made without departing from the spirit and scope of this application. Any person skilled in the art can make their own modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application should be determined by the scope defined in the claims.

Claims

1. A method for segmenting point clouds using lidar, characterized in that, include: Using the rasterized distance view image as input to the segmentation network, the rasterized distance view image is mapped into a high-dimensional embedding to obtain a feature map; Attention is calculated on the feature map to obtain the features corresponding to the feature map; Semantic category prediction is performed on the features to obtain the probability distribution of each grid position in the distance view image for different semantic categories; The probability distribution is used as the output of the segmentation network, and the network output is projected onto a three-dimensional space to complete the semantic segmentation of the lidar point cloud; Before sending the rasterized distance view image to the segmentation network, the method includes: performing data augmentation processing on the rasterized distance view image; the data augmentation processing method consists of four data augmentation methods, namely: mixing point clouds of two frames rasterized into distance view images, performing union of point clouds of two frames rasterized into distance view images, pasting point clouds of two frames rasterized into distance view images, and sliding point cloud on a single frame rasterized into a distance view image. The rasterized distance view image is obtained by rasterizing the lidar point cloud, and the rasterization process includes: Each point in the lidar point cloud is projected onto a grid of a range view image to obtain the point cloud input; the point cloud input is represented by a two-dimensional range view image; and the point cloud input includes the following features: point coordinates, depth, reflection intensity, and mask; wherein, the mask is used to characterize whether each grid position in the point cloud input contains a valid point; Attention is calculated on the feature map using formula (1), which is shown below: （1） in, This is the output of the multi-head attention module; This is a self-attention mechanism; The formula for calculating attention; Represents the normalized exponential function, W represents the dimensions of the head. Q W K W V W O These are the weight matrices for the query Q, key K, value V, and output O in the self-attention mechanism, respectively.

2. The lidar point cloud segmentation method according to claim 1, characterized in that, Before feeding the rasterized distance view image to the segmentation network, the following steps are also included: The rasterized distance view image is cut into several sub-distance view images; The sub-distance view image is used as the input to the segmentation network.

3. A lidar point cloud segmentation system, wherein the system is used to implement the lidar point cloud segmentation method as described in claim 1 or 2, characterized in that, include: The segmentation network and the projection module connected to the segmentation network, wherein the input of the segmentation network is a rasterized distance view image and the output of the segmentation network is the probability distribution of each grid position in the distance view image for different semantic categories; The segmentation network is used to map the rasterized distance view image into a high-dimensional embedding to obtain a feature map; attention is calculated on the feature map to obtain the features corresponding to the feature map; and semantic category prediction is performed on the features to obtain the probability distribution of each grid position in the distance view image for different semantic categories. The projection module is used to project the network output onto a three-dimensional space to complete the semantic segmentation of the lidar point cloud.

4. The lidar point cloud segmentation system according to claim 3, characterized in that, The segmentation network includes a distance embedding module, a segmentation module, and a semantic segmentation head module connected in sequence. Both the distance embedding module and the semantic segmentation head module include several layers of multilayer perceptrons. The segmentation module includes multiple sequentially connected sub-segmentation modules, each of which includes a block embedding module, an addition and normalization module, a forward module, a multi-head attention module, and a block aggregation module. The distance embedding module is used to map the distance view image into a high-dimensional embedding to obtain a feature map; the segmentation module is used to send the feature map into the sub-segmentation module for attention calculation to extract features and obtain features for each stage; the stage corresponds one-to-one with the sub-segmentation module; the semantic segmentation head module is used to perform semantic category prediction using the features of each stage.

5. The lidar point cloud segmentation system according to claim 4, characterized in that, The forward module includes several layers of multilayer perceptrons and activation functions; The forward module is characterized by formula (2): （2） in, Indicates the forward module; Indicates a linear layer; This indicates a residual connection.

6. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the lidar point cloud segmentation method as described in any one of claims 1 to 2.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the lidar point cloud segmentation method according to any one of claims 1 to 2.

Images