Semantic segmentation-oriented laser radar point cloud compression system

Abstract

The invention relates to a semantic segmentation-oriented laser radar point cloud compression system, and belongs to the technical field of automatic driving, and the system comprises a laser radar point cloud projection module which is used for projecting laser radar point cloud data in a three-dimensional space into a depth image, and converting the point cloud data into two-dimensional representation; the nonlinear transformation network is used for carrying out coding processing on the depth image to obtain coding features; the quantization module is used for quantizing the obtained coding features to obtain quantized features; the super-prior transformation network and the super-prior inverse transformation network act on the obtained coding features to generate prior information; the nonlinear inverse transformation network is used for restoring the decoding features into intermediate features for downstream machine vision tasks; and the machine vision task network is used for analyzing and processing the intermediate features transmitted by the nonlinear inverse transformation network to obtain a laser radar point cloud compression result.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving and relates to a laser radar point cloud compression system for semantic segmentation. Background Art

[0002] In recent years, point cloud has become an important data format for autonomous driving systems to provide high-precision object depth information. However, the large amount of data generated by LiDAR (Light Detection And Ranging) is still a problem for data communication. Effectively compressing LiDAR point cloud to meet storage and bandwidth requirements can improve the efficiency of autonomous driving systems. Traditional point cloud coding (PCC) algorithms are mainly divided into two typical PCC architectures: geometry-based PCC method (G-PCC) uses octree structure to compress point cloud; video-based PCC method (V-PCC) uses video codec to compress the projection plane of point cloud. In addition, more and more research has begun to focus on learned PCC methods. Due to the sparsity and irregularity of point cloud, these methods change the original 3D representation of point cloud, such as regular voxels, octrees, depth images, etc. They all take maintaining the signal fidelity of reconstructed point cloud as compression goal, without considering the joint optimization of downstream tasks, which makes the decoded point cloud have obvious loss of perceived accuracy under low bit rate tasks.

[0003] At the same time, with the advancement of big data and artificial intelligence, the data collected by sensors no longer only serves the traditional human visual and auditory systems, but is increasingly used for machine intelligence tasks. However, most deep learning models are very computationally expensive. Under multiple constraints such as size, energy consumption, and cost, edge devices face major challenges in performing machine vision tasks. A common response strategy is to transmit the input data to the server for in-depth analysis. In this mode, the edge device needs to compress the data before transmitting it, and the commonly used codecs are mainly optimized for human vision to reconstruct the input data. However, these traditional codecs are not specifically designed for the needs of machine analysis. Therefore, they often allocate a large number of bits during the encoding process to transmit information that is not critical to the machine task. This results in a trade-off between rate and accuracy that is not as efficient as when using specialized codecs. Especially in the case of poor network conditions or network congestion, the use of these non-specialized codecs may cause a significant increase in the latency of machine vision tasks, thereby affecting the real-time and efficiency of the tasks.

[0004] In recent years, more and more research has been devoted to task-driven image and video compression. These studies aim to jointly optimize the intelligent analysis tasks and compression of images and videos. Existing image / video coding (ICM / VCM) methods designed for machines usually compress images / videos in the original domain of the image / video to serve specific tasks; or compress the intermediate feature representation of the image / video in the feature domain for task analysis. The rapid development of deep learning has made breakthroughs in the field of autonomous driving. The LiDAR perception task converts the LiDAR point cloud into a bird's eye view (BEV) or a range image (RI) representation, converts the irregular LiDAR point cloud into a perceptual two-dimensional representation, and uses 2D to complete various tasks with real-time and high-precision perception capabilities, such as BEV target detection and tracking, RI semantic segmentation, etc. Summary of the invention

[0005] In view of this, an object of the present invention is to provide a LiDAR point cloud compression system for semantic segmentation.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A lidar point cloud compression system for semantic segmentation, comprising:

[0008] LiDAR point cloud projection module: used to project the LiDAR point cloud data in three-dimensional space into a depth image and convert the point cloud data into a two-dimensional representation;

[0009] Nonlinear transformation network: used to encode the depth image and obtain the encoding features;

[0010] Quantization module: used to quantize the obtained coding features to obtain quantitative features;

[0011] Super priori transformation network and super priori inverse transformation network: act on the obtained encoding features to generate prior information;

[0012] Nonlinear inverse transformation network: used to restore the decoded features to intermediate features for downstream machine vision tasks;

[0013] Machine vision task network: used to analyze and process the intermediate features transmitted by the nonlinear inverse transformation network to obtain the lidar point cloud compression results.

[0014] Furthermore, the nonlinear transformation network includes 3 residual downsampling modules and 2 attention modules; each residual downsampling module includes a layer of convolution with a convolution kernel size of 3×3, a downsampling convolution with a convolution kernel size of 3×3 and a step size of 2, and a residual module consisting of two convolutions; the attention module includes three residual modules and a swintransformer block; the nonlinear transformation network takes the depth map obtained by the lidar point cloud projection module as input and converts it into hidden layer features.

[0015] Furthermore, the super-prior transform network obtains super-prior features by passing the hidden layer features output by the nonlinear transform network through the super-prior transform network, and the super-prior features are losslessly encoded into a bit stream. The bit stream is decoded and input into the super-prior inverse transform network to obtain the decoded super-prior features, and the mean and variance of the probability distribution parameters are estimated for arithmetic coding and decoding.

[0016] Furthermore, the prior information includes predicted mean and variance.

[0017] Furthermore, the nonlinear inverse transform network includes three residual upsampling modules and two attention modules; each residual upsampling module includes a layer of convolution with a convolution kernel size of 3×3, an upsampling convolution with a convolution kernel size of 3×3 and a step size of 2, and a residual module consisting of two convolutions; the attention module includes three residual modules and a swintransformer block; the nonlinear inverse transform network converts the decoded feature y hat Restore to the original depth map size, output high-dimensional intermediate features denoted as x hat .

[0018] Furthermore, the machine vision task network inputs the intermediate features output by the decoder into the machine vision task network to obtain the result of the specified machine task.

[0019] Furthermore, the loss function of the machine vision task network during the training process is expressed as:

[0020] Loss = D(f(x hat ),G gt )+λ×R

[0021] Where f(·) represents the machine vision task network, G gt represents the target value of the machine task, and D(·) represents the predicted value f(x hat ) and target value G gt , R represents the number of bits consumed in the transmission process, and λ is a hyperparameter.

[0022] The beneficial effect of the present invention is that the present invention converts irregular LiDAR point clouds into regular two-dimensional representations of depth images, takes the depth images as compression objects, proposes an end-to-end feature compression framework, and decodes them into intermediate features before executing downstream tasks, and then inputs the intermediate features into the analysis network of downstream tasks to achieve efficient LiDAR point cloud feature compression. Under low bit rates, downstream machine tasks can still maintain high-precision perception capabilities. Specifically, the present invention first converts irregular LiDAR point clouds into regular two-dimensional representation depth maps by combining specific downstream tasks, takes the depth maps as compression objects, proposes an end-to-end feature compression framework (LiDAR Point Cloud Coding for Machine, LPCCM), and decodes them into intermediate features before executing downstream tasks, and then inputs the intermediate features into the analysis network of downstream tasks to achieve efficient LiDAR point cloud feature compression. Compared with traditional compression methods oriented to geometric fidelity, this method can still maintain high-precision perception capabilities for downstream machine tasks under low bit rates.

[0023] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0025] Figure 1 This is a framework diagram of the LiDAR point cloud compression system for semantic segmentation according to the present invention;

[0026] Figure 2 This is a specific structural diagram of the laser radar point cloud compression system for semantic segmentation according to the present invention;

[0027] Figure 3 It is a structural diagram of residual downsampling and upsampling in the network structure of the present invention;

[0028] Figure 4 is a structural diagram of the attention module of the present invention;

[0029] Figure 5 This is a comparison chart of the present invention and other point cloud compression algorithms. DETAILED DESCRIPTION

[0030] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0031] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0032] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0033] like Figure 1 As shown, the present invention provides a lidar point cloud compression system for semantic segmentation, which converts the lidar point cloud into a regular two-dimensional depth map representation, takes the depth map as the compression object, and decodes it into intermediate features before executing downstream tasks, and then inputs the intermediate features into the analysis network of the downstream task to execute the downstream task.

[0034] like Figure 2 As shown, the system includes a lidar point cloud projection module, a nonlinear transformation network, a quantization module, a hyper-prior transformation network, a hyper-prior inverse transformation network, a nonlinear inverse transformation network, and a machine vision task network.

[0035] The LiDAR point cloud projection module projects the LiDAR point cloud data in three-dimensional space into a depth image and converts the point cloud data into a two-dimensional representation. Specifically, given a LiDAR point cloud data, a LiDAR point cloud represented by Cartesian coordinates is converted into spherical coordinates to calculate the distance to the LiDAR sensor. The distance value is assigned to the corresponding image pixel, thereby obtaining the depth image representation of the LiDAR. First, in the Cartesian coordinate system, the coordinates of each three-dimensional coordinate point are marked as (x, y, z). Then, each point is converted to the spherical coordinate system through a spherical coordinate transformation, which is defined as follows:

[0036]

[0037] Its coordinates are represented as Next, project the point cloud to the specified resolution of C×H×W:

[0038]

[0039] where f v =|f down |+|f up |, is the vertical field of view of the lidar sensor, and (ρ, x, y, z) is loaded into the corresponding pixel position to obtain the depth map, and pixels without points are filled with 0; the final two-dimensional depth map representation of the lidar point cloud is denoted by x.

[0040] The nonlinear transformation network encodes the depth image to obtain the encoded features; it contains 3 residual downsampling modules and two attention modules. The nonlinear transformation network will obtain compact encoded features after three downsamplings; each residual downsampling module contains a layer of convolution with a convolution kernel size of 3×3, a downsampling convolution with a convolution kernel size of 3×3 and a stride of 2, and a residual module consisting of two convolutions. The attention module contains three residual modules and a swintransformer block, such as Figure 2 and Figure 3 As shown in Figure 2, the depth map obtained by the LiDAR point cloud projection module is used as the object, and the depth map is used as the input of the nonlinear transformation network to convert it into hidden layer features.

[0041] The quantization module quantizes the obtained coding features to obtain quantized features. Specifically, the coding feature y is quantized to obtain the quantized feature y hat .

[0042] The super-prior transform network and the super-prior inverse transform network act on the obtained coding features to generate prior information; the super-prior transform network and the super-prior inverse transform network act on the obtained coding features to generate prior information, which includes the predicted mean and variance. The super-prior transform network passes the hidden layer features output by the nonlinear transform network through the super-prior transform network to obtain the super-prior features, which are losslessly encoded into a bit stream, and the bit stream is decoded and input into the super-prior inverse transform network to obtain the decoded super-prior features, and the probability distribution parameter mean and variance are estimated for arithmetic coding and decoding. Specifically including: super-prior feature z hat Get the predicted mean and variance (μ, σ), and quantize the encoded feature y hat Perform lossless arithmetic coding to obtain a bit stream, transmit the bit stream to the decoding end, and restore the bit stream to the decoding feature y through lossless arithmetic decoding hat .

[0043] The nonlinear inverse transformation network restores the decoded features to intermediate features for downstream machine vision tasks (without reconstruction into point clouds). hat Restore to the original depth map size, output high-dimensional intermediate features denoted as x hat The nonlinear inverse transformation network includes three residual upsampling modules and two attention modules corresponding to the nonlinear transformation network. The nonlinear inverse transformation network will obtain the intermediate features of the output through three upsamplings; each residual upsampling module includes a layer of convolution with a convolution kernel size of 3×3, an upsampling convolution with a convolution kernel size of 3×3 and a stride of 2, and a residual module consisting of two convolutions. Figure 4 As shown, the attention module contains three residual modules and one swintransformer block.

[0044] Machine vision task network. It is used to receive the feature information required by the machine vision task network, analyze and process it, and obtain the result of the specified machine task. hat Directly input into the machine vision task network. The machine vision task network selected in this embodiment is the Rangenet++ semantic segmentation network, a lidar point cloud compression system for semantic segmentation. The loss function in the training process is expressed as:

[0045] Loss = D(f(x hat ),G gt )+λ×R

[0046] Where f(·) represents the machine vision task network, G gt represents the target value of the machine task, and D(·) represents the predicted value f(x hat ) and target value G gt , R represents the number of bits consumed in the transmission process, and λ is a hyperparameter.

[0047] like Figure 5 The figure shows the comparison test between the present invention and the traditional GPCC point cloud compression method and a deep learning point cloud lossless compression algorithm OctreeAttention on the Segmatic KITTI dataset, and the evaluation index is the average intersection over union ratio of semantic segmentation. The comparison method is to compress the lidar point cloud data using GPCC and OctreeAttention and reconstruct it into point cloud, and then segment it through the Rangenet++ semantic segmentation network. The results show that when facing downstream tasks, the present invention has higher compression efficiency than the point cloud compression method that pursues geometric fidelity.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A LiDAR point cloud compression system for semantic segmentation, characterized by: include: LiDAR point cloud projection module: used to project the LiDAR point cloud data in three-dimensional space into a depth image and convert the point cloud data into a two-dimensional representation; Nonlinear transformation network: used to encode the depth image and obtain the encoding features; Quantization module: used to quantize the obtained coding features to obtain quantitative features; Super priori transformation network and super priori inverse transformation network: act on the obtained encoding features to generate prior information; Nonlinear inverse transformation network: used to restore the decoded features to intermediate features for downstream machine vision tasks; Machine vision task network: used to analyze and process the intermediate features transmitted by the nonlinear inverse transformation network to obtain the lidar point cloud compression results.

2. The LiDAR point cloud compression system for semantic segmentation according to claim 1, characterized in that: The nonlinear transformation network includes 3 residual downsampling modules and 2 attention modules; each residual downsampling module includes a layer of convolution with a convolution kernel size of 3×3, a downsampling convolution with a convolution kernel size of 3×3 and a step size of 2, and a residual module consisting of two convolutions; the attention module includes three residual modules and a swin transformer block; the nonlinear transformation network takes the depth map obtained by the lidar point cloud projection module as input and converts it into hidden layer features.

3. The LiDAR point cloud compression system for semantic segmentation according to claim 1, characterized in that: The super-prior transformation network obtains super-prior features by passing the hidden layer features output by the nonlinear transformation network through the super-prior transformation network. The super-prior features are losslessly encoded into a bit stream, and the bit stream is decoded and input into the super-prior inverse transformation network to obtain the decoded super-prior features, and the mean and variance of the probability distribution parameters are estimated for arithmetic coding and decoding.

4. The LiDAR point cloud compression system for semantic segmentation according to claim 3, characterized in that: The prior information includes the predicted mean and variance.

5. The LiDAR point cloud compression system for semantic segmentation according to claim 1, characterized in that: The nonlinear inverse transform network includes three residual upsampling modules and two attention modules; each residual upsampling module includes a layer of convolution with a convolution kernel size of 3×3, an upsampling convolution with a convolution kernel size of 3×3 and a step size of 2, and a residual module consisting of two convolutions; the attention module includes three residual modules and a swin transformer block; the nonlinear inverse transform network converts the decoded feature y hat Restore to the original depth map size, output high-dimensional intermediate features denoted as x hat .

6. The LiDAR point cloud compression system for semantic segmentation according to claim 1, characterized in that: The machine vision task network inputs the intermediate features output by the decoder into the machine vision task network to obtain the result of the specified machine task.

7. The LiDAR point cloud compression system for semantic segmentation according to claim 1, characterized in that: The loss function of the machine vision task network during the training process is expressed as: Loss=D(f(x hat ),G gt )+λ×R Where f(·) represents the machine vision task network, G gt represents the target value of the machine task, and D(·) represents the predicted value f(x hat ) and target value G gt , R represents the number of bits consumed in the transmission process, and λ is a hyperparameter.

Images