A LiDAR point cloud compression system for semantic segmentation

Abstract

The present invention relates to a lidar point cloud compression system for semantic segmentation, which belongs to the field of autonomous driving technology and includes: a lidar point cloud projection module: used to project lidar point cloud data in three-dimensional space into a depth image, and convert the point cloud data into a two-dimensional representation; a nonlinear transformation network: used to encode the depth image to obtain encoding features; a quantization module: used to quantize the obtained encoding features to obtain quantized features; a super-prior transformation network and a super-prior inverse transformation network: acting on the obtained encoding features to generate prior information; a nonlinear inverse transformation network: used to restore the decoded features to intermediate features for downstream machine vision tasks; and a machine vision task network: used to analyze and process the intermediate features transmitted by the nonlinear inverse transformation network to obtain lidar point cloud compression results.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving and relates to a lidar point cloud compression system for semantic segmentation. Background Art

[0002] In recent years, point clouds have become an essential data format for providing high-precision object depth information in autonomous driving systems. However, the massive amount of data generated by LiDAR (Light Detection and Ranging) sensors remains a data communication challenge. Effectively compressing LiDAR point clouds to meet storage and bandwidth requirements can improve the efficiency of autonomous driving systems. Traditional point cloud coding (PCC) algorithms are mainly divided into two typical PCC architectures: geometry-based PCC methods (G-PCC) utilize an octree structure to compress point clouds, and video-based PCC methods (V-PCC) utilize video codecs to compress the projected planes of point clouds. Furthermore, a growing body of research has focused on learned PCC methods. Due to the sparsity and irregularity of point clouds, these methods modify the original 3D representation of point clouds, such as regular voxels, octrees, and depth images. These methods all focus on maintaining the signal fidelity of the reconstructed point clouds as compression without considering joint optimization of downstream tasks. This results in a significant loss of perceptual accuracy in the decoded point clouds for low-bitrate tasks.

[0003] With the advancement of big data and artificial intelligence, data collected by sensors is no longer solely used for traditional human vision and hearing systems; it is increasingly being used for machine intelligence tasks. However, most deep learning models are computationally expensive. Due to multiple constraints such as size, energy consumption, and cost, edge devices face significant challenges in performing machine vision tasks. A common strategy is to transmit input data to a server for in-depth analysis. In this model, edge devices compress data before transmission, using codecs optimized for human vision to reconstruct the input data. However, these traditional codecs are not specifically designed for the needs of machine analysis. As a result, they often allocate a large number of bits during the encoding process to transmit information that is not critical to the machine task. This results in a less efficient trade-off between speed and accuracy than using specialized codecs. Especially in poor or congested network conditions, the use of these non-specialized codecs can significantly increase latency in machine vision tasks, impacting both real-time performance and efficiency.

[0004] In recent years, a growing number of studies have been devoted to task-driven image and video compression. These studies aim to jointly optimize intelligent image and video analysis tasks with compression. Existing machine-designed image / video coding (ICM / VCM) methods typically compress images / videos in their original domain to serve specific tasks, or compress intermediate feature representations of images / videos in the feature domain for task analysis. The rapid development of deep learning has enabled breakthroughs in the field of autonomous driving. LiDAR perception tasks convert LiDAR point clouds into bird's-eye view (BEV) or range image (RI) representations, converting irregular LiDAR point clouds into perceptual two-dimensional representations, and using 2D forms to complete various tasks with real-time, high-precision perception capabilities, such as BEV target detection and tracking and RI semantic segmentation. Summary of the Invention

[0005] In view of this, the object of the present invention is to provide a lidar point cloud compression system for semantic segmentation.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A lidar point cloud compression system for semantic segmentation, comprising:

[0008] LiDAR point cloud projection module: used to project the LiDAR point cloud data in three-dimensional space into a depth image and convert the point cloud data into a two-dimensional representation;

[0009] Nonlinear transformation network: used to encode the depth image and obtain the encoding features;

[0010] Quantization module: used to quantize the obtained coding features to obtain quantitative features;

[0011] Super priori transform network and super priori inverse transform network: act on the obtained encoding features to generate prior information;

[0012] Nonlinear inverse transformation network: used to restore decoded features to intermediate features for downstream machine vision tasks;

[0013] Machine vision task network: used to analyze and process the intermediate features transmitted by the nonlinear inverse transformation network to obtain the lidar point cloud compression results.

[0014] Furthermore, the nonlinear transformation network includes 3 residual downsampling modules and 2 attention modules; each residual downsampling module includes a layer of convolution with a convolution kernel size of 3×3, a downsampling convolution with a convolution kernel size of 3×3 and a stride of 2, and a residual module consisting of two convolutions; the attention module includes three residual modules and a swintransformer block; the nonlinear transformation network takes the depth map obtained by the lidar point cloud projection module as input and converts it into hidden layer features.

[0015] Furthermore, the super-prior transformation network obtains super-prior features by passing the hidden layer features output by the nonlinear transformation network through the super-prior transformation network. The super-prior features are losslessly encoded into a bit stream, and the bit stream is decoded and input into the super-prior inverse transformation network to obtain the decoded super-prior features, and the mean and variance of the probability distribution parameters are estimated for arithmetic coding and decoding.

[0016] Furthermore, the prior information includes predicted mean and variance.

[0017] Furthermore, the nonlinear inverse transform network includes three residual upsampling modules and two attention modules; each residual upsampling module includes a layer of convolution with a convolution kernel size of 3×3, an upsampling convolution with a convolution kernel size of 3×3 and a step size of 2, and a residual module consisting of two convolutions; the attention module includes three residual modules and a swintransformer block; the nonlinear inverse transform network will decode the feature y hat Restore to the original depth map size and output high-dimensional intermediate features as x hat .

[0018] Furthermore, the machine vision task network inputs the intermediate features output by the decoder into the machine vision task network to obtain the result of the specified machine task.

[0019] Furthermore, the loss function of the machine vision task network during the training process is expressed as:

[0020] Loss=D(f(x hat ),G gt )+λ×R

[0021] where f(·) represents the machine vision task network, G gt represents the target value of the machine task, D(·) represents the predicted value f(x hat ) and target value G gt The loss between them, R represents the number of bits consumed in the transmission process, and λ is a hyperparameter.

[0022] The beneficial effect of the present invention is that the present invention converts irregular LiDAR point clouds into regular two-dimensional representations of depth images, takes the depth images as the compression objects, proposes an end-to-end feature compression framework, and decodes them into intermediate features before executing downstream tasks, and then inputs the intermediate features into the analysis network of the downstream tasks to achieve efficient LiDAR point cloud feature compression. Under low bit rates, the downstream machine tasks can still maintain high-precision perception capabilities. Specifically, the present invention first converts irregular LiDAR point clouds into regular two-dimensional representation depth maps by combining specific downstream tasks, takes the depth maps as the compression objects, proposes an end-to-end feature compression framework (LiDAR Point Cloud Coding for Machine, LPCCM), and decodes them into intermediate features before executing downstream tasks, and then inputs the intermediate features into the analysis network of the downstream tasks to achieve efficient LiDAR point cloud feature compression. Compared with traditional geometric fidelity-oriented compression methods, this method can maintain high-precision perception capabilities for downstream machine tasks under low bit rates.

[0023] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0025] Figure 1 This is a framework diagram of the lidar point cloud compression system for semantic segmentation according to the present invention;

[0026] Figure 2 This is a specific structural diagram of the lidar point cloud compression system for semantic segmentation according to the present invention;

[0027] Figure 3 This is a structural diagram of residual downsampling and upsampling in the network structure of the present invention;

[0028] Figure 4 This is a structural diagram of the attention module of the present invention;

[0029] Figure 5 This is a comparison chart of the present invention and other point cloud compression algorithms. DETAILED DESCRIPTION

[0030] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0031] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0032] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention.

[0033] like Figure 1 As shown, the present invention provides a lidar point cloud compression system for semantic segmentation, which converts the lidar point cloud into a regular two-dimensional depth map representation, uses the depth map as the compression object, and decodes it into intermediate features before executing downstream tasks, and then inputs the intermediate features into the analysis network of the downstream task to execute the downstream task.

[0034] like Figure 2 As shown, the system includes a lidar point cloud projection module, a nonlinear transformation network, a quantization module, a super priori transformation network, a super priori inverse transformation network, a nonlinear inverse transformation network, and a machine vision task network.

[0035] The LiDAR point cloud projection module projects LiDAR point cloud data in three-dimensional space into a depth image, converting the point cloud data into a two-dimensional representation. Specifically, given a LiDAR point cloud, a Cartesian coordinate representation of the LiDAR point cloud is converted into spherical coordinates to calculate the distance to the LiDAR sensor. The distance value is assigned to the corresponding image pixel, thereby obtaining the LiDAR depth image representation. First, in the Cartesian coordinate system, the coordinates of each three-dimensional coordinate point are labeled (x, y, z). Then, each point is converted to the spherical coordinate system through a spherical coordinate transformation, which is defined as follows:

[0036]

[0037] Its coordinate representation is Next, project the point cloud to the specified resolution of C×H×W:

[0038]

[0039] where f v =|f down |+|f up |, is the vertical field of view of the lidar sensor, and (ρ, x, y, z) is loaded into the corresponding pixel position to obtain the depth map. Pixels without points are filled with 0; the final two-dimensional depth map representation of the lidar point cloud is recorded as x.

[0040] The nonlinear transformation network encodes the depth image to obtain the encoded features; it contains three residual downsampling modules and two attention modules. The nonlinear transformation network will obtain compact encoded features after three downsamplings; each residual downsampling module contains a layer of convolution with a 3×3 convolution kernel size, a downsampling convolution with a convolution kernel size of 3×3 and a stride of 2, and a residual module consisting of two convolutions. The attention module contains three residual modules and a swintransformer block, such as Figure 2 and Figure 3 As shown in Figure 2, the depth map obtained by the LiDAR point cloud projection module is used as the object and the depth map is used as the input of the nonlinear transformation network to convert it into hidden layer features.

[0041] The quantization module quantizes the obtained coding features to obtain quantitative features. Specifically, the coding features y are quantized to obtain quantitative features y hat .

[0042] The super-prior transform network and the super-prior inverse transform network act on the obtained coding features to generate prior information; the super-prior transform network and the super-prior inverse transform network act on the obtained coding features to generate prior information, which includes the predicted mean and variance. The super-prior transform network passes the hidden layer features output by the nonlinear transform network through the super-prior transform network to obtain the super-prior features. The super-prior features are losslessly encoded into a bit stream, and the bit stream is decoded and input into the super-prior inverse transform network to obtain the decoded super-prior features, and the probability distribution parameter mean and variance are estimated for arithmetic coding and decoding. Specifically including: super-prior feature z hat Get the predicted mean and variance (μ, σ), and quantize the encoded feature y hat Perform lossless arithmetic coding to obtain the bit stream, transmit the bit stream to the decoding end, and restore the bit stream to the decoding feature y through lossless arithmetic decoding hat .

[0043] The nonlinear inverse transformation network restores the decoded features to intermediate features for downstream machine vision tasks (without the need to reconstruct them into point clouds). hat Restore to the original depth map size and output high-dimensional intermediate features as x hat The nonlinear inverse transformation network includes three residual upsampling modules and two attention modules corresponding to the nonlinear transformation network. The nonlinear inverse transformation network will obtain the intermediate features of the output through three upsamplings; each residual upsampling module contains a layer of convolution with a convolution kernel size of 3×3, an upsampling convolution with a convolution kernel size of 3×3 and a stride of 2, and a residual module consisting of two convolutions. Figure 4 As shown, the attention module contains three residual modules and a swintransformer block.

[0044] Machine vision task network. It is used to receive the feature information required by the machine vision task network, perform analysis and processing, and obtain the result of the specified machine task. The intermediate feature x output by the decoder is converted to hat Directly input into the machine vision task network. The machine vision task network selected in this embodiment is the Rangenet++ semantic segmentation network. The loss function of the lidar point cloud compression system for semantic segmentation is expressed as:

[0045] Loss=D(f(x hat ),G gt )+λ×R

[0046] where f(·) represents the machine vision task network, G gt represents the target value of the machine task, D(·) represents the predicted value f(x hat ) and target value G gt The loss between them, R represents the number of bits consumed in the transmission process, and λ is a hyperparameter.

[0047] like Figure 5 Figure 2 shows a comparative test of the present invention on the Segmatic KITTI dataset, using the traditional GPCC point cloud compression method and OctreeAttention, a deep learning-based lossless point cloud compression algorithm. The evaluation metric is the average intersection over union (IoU) of semantic segmentation. The comparison method uses GPCC and OctreeAttention to compress and reconstruct LiDAR point cloud data into point clouds, followed by segmentation using a Rangenet++ semantic segmentation network. The results demonstrate that the present invention achieves higher compression efficiency for downstream tasks than point cloud compression methods that prioritize geometric fidelity.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A LiDAR point cloud compression system for semantic segmentation, characterized by: include: LiDAR point cloud projection module: used to project the LiDAR point cloud data in three-dimensional space into a depth image and convert the point cloud data into a two-dimensional representation; Nonlinear transformation network: used to encode the depth image and obtain the encoding features; Quantization module: used to quantize the obtained coding features to obtain quantitative features; Super priori transform network and super priori inverse transform network: act on the obtained encoding features to generate prior information; Nonlinear inverse transformation network: used to restore decoded features to intermediate features for downstream machine vision tasks; Machine vision task network: used to analyze and process the intermediate features transmitted by the nonlinear inverse transform network to obtain the lidar point cloud compression results; The super-prior transformation network passes the hidden layer features output by the nonlinear transformation network through the super-prior transformation network to obtain super-prior features, the super-prior features are losslessly encoded into a bit stream, the bit stream is decoded and input into the super-prior inverse transformation network to obtain decoded super-prior features, and the mean and variance of the probability distribution parameters are estimated for arithmetic coding and decoding; the prior information includes the predicted mean and variance; The nonlinear inverse transform network includes three residual upsampling modules and two attention modules; each residual upsampling module includes a layer of convolution with a 3×3 convolution kernel size, an upsampling convolution with a convolution kernel size of 3×3 and a step size of 2, and a residual module consisting of two convolutions; the attention module includes three residual modules and a swin transformer block; the nonlinear inverse transform network will decode the features Restore to the original depth map size and output high-dimensional intermediate features as .

2. The LiDAR point cloud compression system for semantic segmentation according to claim 1, characterized in that: The nonlinear transformation network includes three residual downsampling modules and two attention modules; each residual downsampling module includes a layer of convolution with a 3×3 convolution kernel size, a downsampling convolution with a convolution kernel size of 3×3 and a stride of 2, and a residual module consisting of two convolutions; the attention module includes three residual modules and a swin transformer block; the nonlinear transformation network takes the depth map obtained by the lidar point cloud projection module as input and converts it into hidden layer features.

3. The LiDAR point cloud compression system for semantic segmentation according to claim 1, characterized in that: The machine vision task network inputs the intermediate features output by the decoder into the machine vision task network to obtain the result of the specified machine task.

4. The LiDAR point cloud compression system for semantic segmentation according to claim 1, characterized in that: The loss function of the machine vision task network during the training process is expressed as: in represents the machine vision task network, represents the target value of the machine task, Represents the predicted value and target value The loss between Indicates the number of bits consumed during the transmission process. is a hyperparameter.

Images