Point cloud processing method and device, electronic equipment and storage medium

Abstract

The invention provides a point cloud processing method and device based on aerial view representation, electronic equipment and a storage medium. The point cloud processing method comprises the following steps: 1) inputting a three-dimensional point cloud grid set V, and extracting features of point cloud points in three-dimensional point cloud grids as grid features; 2) features among three-dimensional point cloud grids in a neural network fusion set V are utilized, and the features of the three-dimensional point cloud grids are converted into bird's-eye representation of a two-dimensional space; and 3) encoding the bird's-eye view representation of the two-dimensional space to form a code stream. Furthermore, the code stream is analyzed to obtain the bird's-eye view representation of the reconstructed two-dimensional space, and the bird's-eye view representation of the reconstructed two-dimensional space is decoded to obtain the bird's-eye view representation of the two-dimensional space representing the three-dimensional point cloud grid information. According to the method, the aerial view representation codes of the two-dimensional space can be compressed by utilizing the aerial view representation codes of the two-dimensional space, the aerial view representation of the two-dimensional space can be decoded and reconstructed by utilizing the aerial view representation decoding method of the two-dimensional space, and finally the aerial view representation is applied to a machine intelligent task.

Description

Technical Field

[0001] The patent of this invention belongs to the field of point cloud coding, and specifically relates to a point cloud processing method, device, electronic device and storage medium based on bird's-eye view representation. Background Art

[0002] In recent years, point clouds have gradually become an important data form that can provide high-precision depth information of distant objects in autonomous driving systems. However, the huge amount of data generated by LiDAR point clouds is still a challenge for data communication and storage. Efficiently compressing LiDAR point clouds to meet the requirements of communication bandwidth can effectively improve the efficiency of autonomous driving systems.

[0003] Traditional point cloud compression algorithms have developed rapidly under the efforts of the International Moving Picture Experts Group (MPEG), and are mainly divided into two typical point cloud compression architectures, namely, geometry-based Point Cloud Compression (G-PCC) and video-based Point Cloud Compression (V-PCC). Among them, G-PCC uses the octree model to efficiently compress point cloud data, while V-PCC is implemented by encoding the projection from three-dimensional to two-dimensional and the video encoder. At the same time, many studies have begun to focus on learning-based point cloud compression methods. Due to the sparsity and irregularity of point cloud data, many learning-based methods have changed the original three-dimensional representation of point clouds, such as representing point clouds as regular voxels, octree expressions, and converting them into expressions of depth images, etc., and using the neighborhood relationships of these regular expressions to reconstruct the signal fidelity of the point cloud as the optimization target to compress the encoded point cloud.

[0004] Deep learning technology has promoted the rapid development of autonomous driving perception technology. The single-vehicle intelligent perception system converts the lidar point cloud into a bird's-eye-view (BEV) representation, showing real-time and high-precision perception capabilities, and has developed a considerable number of perception tasks under BEV, such as BEV target detection and tracking and BEV map segmentation. At the same time, in order to compensate for the limitations of the limited perception range of a single vehicle, the collaborative perception method effectively expands the perception range of the autonomous driving system by transmitting intermediate features of the point cloud between vehicles and between vehicles and roadside facilities.

[0005] In the field of image and video compression, more and more research has begun to focus on image / video compression (ICM / VCM) for machine intelligence tasks. They are optimized for machine vision and have achieved remarkable results. Existing ICM / VCM methods are divided into two categories, either compressing the image / video in the original domain and sending it to the machine for consumption, or compressing the intermediate features in the image / video feature domain for task analysis. To date, the method of compressing intermediate features has become the mainstream branch of ICM / VCM. They take task loss and bit rate loss as optimization goals, and have exceeded the latest VVC standard in multiple intelligent analysis tasks. Existing point cloud coding methods all use the signal fidelity of the reconstructed point cloud as an optimization goal, and no point cloud coding methods for the accuracy of autonomous driving perception tasks have been studied, resulting in the reconstructed point cloud of the existing point cloud encoding and decoding methods still suffering from obvious task accuracy loss.

[0006] Inspired by ICM / VCM research and combined with classic LiDAR point cloud perception methods, point clouds are converted to bird’s-eye view Figure 2 dimensional representation, and encode the two-dimensional features with the help of image coding technology, and finally use the decoded and restored bird's-eye view features to complete the subsequent task prediction, consuming a lower bit rate at the same machine task accuracy. Summary of the invention

[0007] In view of the defects of the above-mentioned prior arts, the present invention proposes a point cloud processing method and device based on bird's-eye view representation, which can convert a three-dimensional point cloud raster into a bird's-eye view representation of a two-dimensional space. According to the characteristics of the bird's-eye view representation, the bird's-eye view representation encoding of the two-dimensional space is used to compress the bird's-eye view representation encoding of the two-dimensional space, and the bird's-eye view representation decoding method of the two-dimensional space is used to decode and reconstruct the bird's-eye view representation of the two-dimensional space, which is finally applied to machine intelligence tasks.

[0008] To this end, the first object of the present invention is to provide a point cloud coding processing method, which includes:

[0009] 1) Input a three-dimensional point cloud grid set V, and extract the features of the point cloud points in the three-dimensional point cloud grid as grid features;

[0010] 2) Using a neural network to fuse the features between the three-dimensional point cloud grids in the set V, the three-dimensional point cloud grid features are converted into a bird's-eye view representation of the two-dimensional space;

[0011] 3) Encode the bird's-eye view representation of the two-dimensional space to form a code stream.

[0012] Furthermore, extracting features of point cloud points in the three-dimensional point cloud grid as grid features includes:

[0013] For the 3D point cloud grid v in the set V i , process and fuse 3D point cloud grid v iThe geometric and attribute information of the inner point cloud points is used to represent the fused features as a three-dimensional point cloud grid v i feature information.

[0014] Furthermore, the encoding process of the bird's-eye view representation of the two-dimensional space to form a code stream includes the following steps:

[0015] 1) Using a neural network-based image coding method to downscale the bird's-eye view representation of the two-dimensional space to obtain a downscaled bird's-eye view representation of the two-dimensional space;

[0016] 2) Perform entropy coding on the downscaled bird's-eye view representation of the two-dimensional space to form a bit stream.

[0017] A second object of the present invention is to provide a point cloud decoding processing method, which comprises:

[0018] 1) Parse the bitstream to obtain a bird's-eye view representation of the reconstructed two-dimensional space;

[0019] 2) Decoding the reconstructed two-dimensional bird's-eye view representation to obtain a two-dimensional bird's-eye view representation representing the three-dimensional point cloud grid information.

[0020] Furthermore, the reconstructed bird's-eye view representation of the two-dimensional space is decoded to obtain a bird's-eye view representation of the two-dimensional space representing the three-dimensional point cloud grid information. The decoding process includes:

[0021] A neural network-based image decoding method is used to upscale the quantized two-dimensional spatial downscaled bird's-eye view representation to obtain a two-dimensional spatial bird's-eye view representation representing the three-dimensional point cloud raster information.

[0022] A third object of the present invention is to provide a point cloud coding processing device, comprising:

[0023] A point cloud grid feature extraction module is used to extract the features of the point cloud points in the input three-dimensional point cloud grid as grid features;

[0024] A 3D point cloud grid transformation 2D bird's-eye view representation module is used to fuse the features between 3D point cloud grids using a neural network and transform the 3D point cloud grid features into a 2D bird's-eye view representation.

[0025] The two-dimensional space bird's-eye view representation encoding module is used to encode the two-dimensional space bird's-eye view representation to form a code stream.

[0026] A fourth object of the present invention is to provide a point cloud decoding processing device, comprising:

[0027] A code stream analysis module, used to analyze the code stream to obtain a bird's-eye view representation of the reconstructed two-dimensional space;

[0028] The two-dimensional space bird's-eye view representation decoding module is used to decode the reconstructed two-dimensional space bird's-eye view representation to obtain the two-dimensional space bird's-eye view representation representing the three-dimensional point cloud grid information.

[0029] A fifth object of the present invention is to provide a point cloud processing electronic device, comprising:

[0030] Memory for storing computer programs;

[0031] The processor is used to execute the program stored in the memory to implement the steps of any one of the above-mentioned point cloud processing methods.

[0032] A sixth object of the present invention is to provide a point cloud processing storage medium, comprising:

[0033] The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any one of the above-mentioned point cloud processing methods are implemented.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The present invention proposes a point cloud processing method based on bird's-eye view, converting a three-dimensional point cloud into a bird's-eye view representation of a two-dimensional space, and designing a codec for the bird's-eye view representation of a two-dimensional space to obtain a more compact feature representation, code compression, and reconstruct the bird's-eye view representation by fusing compression and enhanced compression. The beneficial effects of the present invention are as follows:

[0036] (1) Compared with the existing image encoding and decoding methods, the encoding and decoding method of the bird's-eye view representation of the two-dimensional space of the present invention can more fully extract the feature information of the sparse bird's-eye view image and improve the coding efficiency of the bit rate-task accuracy, that is, under the condition of the same task accuracy, a lower bit rate size is consumed.

[0037] (2) Compared with the existing point cloud encoding and decoding methods, the present invention converts the three-dimensional point cloud into a two-dimensional bird's-eye view representation plane, encodes and compresses the two-dimensional bird's-eye view representation, and applies it to machine intelligence tasks to greatly improve the coding efficiency for the accuracy of machine intelligence tasks. The experimental results show that our method can greatly improve the coding efficiency compared with the traditional point cloud compression coding method G-PCC under the two machine intelligence task accuracy indicators. In the ablation experiment, the coding efficiency of our two-dimensional bird's-eye view representation codec is also better than the existing intelligent image coding method. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a structural flow chart of a point cloud coding processing method based on bird's-eye view representation;

[0039] Figure 2Schematic diagram of a columnar grid and a voxel grid in an embodiment;

[0040] Figure 3 A schematic diagram showing a bird's-eye view representation of a two-dimensional space formed by a grid in an embodiment;

[0041] Figure 4 This is a schematic diagram of the structure of a multi-scale sparse feature fusion compression or fusion reconstruction module in an embodiment;

[0042] Figure 5 It is a structural diagram of an enhanced ConvNeXt module of a multi-scale sparse feature extraction module in an embodiment;

[0043] Figure 6 This is a schematic diagram of the structure of a sparse feature attention enhancement compression or enhancement reconstruction module in an embodiment;

[0044] Figure 7 Schematic diagram of the visual attention network structure in the embodiment;

[0045] Figure 8 This is a schematic diagram of the structure of a large core attention block in the embodiment;

[0046] Fig. 9 It is a schematic diagram of the structure of a downsampling module of a bird's-eye view representation encoder of a two-dimensional space in an embodiment;

[0047] Fig.10 This is a schematic diagram of the structure of the entropy coding module in the embodiment;

[0048] Fig.11 It is a structural flow chart of a point cloud decoding processing method based on bird's-eye view representation;

[0049] Fig.12 Schematic diagram of the structure of an up-sampling module of a bird's-eye view representation decoder of a two-dimensional space in an embodiment;

[0050] Fig.13 A schematic diagram of a region proposal network represented by a bird's-eye view in an embodiment;

[0051] Fig.14 It is a structural flow chart of a point cloud coding processing device based on bird's-eye view representation;

[0052] Fig.15 A structural flow chart of a point cloud decoding and processing device based on bird's-eye view representation DETAILED DESCRIPTION

[0053] In order to further understand the present invention, preferred embodiments of the present invention are described below in conjunction with examples. However, it should be understood that these descriptions are only for further illustrating the features and advantages of the present invention, rather than limiting the claims of the present invention.

[0054] Below, the present invention is further explained in conjunction with the accompanying drawings:

[0055] Example 1

[0056] The embodiment of the present invention provides a point cloud processing method based on bird's-eye view representation, such as Figure 1 As shown, the specific steps include:

[0057] S11. Obtain an input three-dimensional point cloud grid set V, and extract features of point cloud points in the three-dimensional point cloud grid as three-dimensional point cloud grid features. The following is a specific example:

[0058] Since the original point cloud is irregularly distributed, the point cloud can be represented regularly through a three-dimensional point cloud grid. The three-dimensional point cloud grid includes a columnar grid or a voxel grid. The set of three-dimensional columnar grids or voxel grids is uniformly represented by V and is marked as V:{v1,v2,…,v i ,…,v n}, each three-dimensional grid is represented by v i :{p1,p2,…,p i ,…,p m},i∈1~n, where p j :{x j ,y j ,z j ,r j ,g j ,b j ,reflectance j}, j∈1~m, represents the point cloud points in each grid, where x j ,y j ,z j ,r j ,g j ,b j ,reflectance j Represent the point cloud p j The x-coordinate, y-coordinate, and z-coordinate of the point's geometric information and the red component, green component, blue component, and reflectivity of its attribute information.

[0059] For each 3D grid v in the set V i , process and fuse 3D point cloud grid v i The geometric and attribute information of the inner point cloud points is used to represent the fused features as a three-dimensional point cloud grid v i feature information. Figure 2 Two specific examples are shown. Figure 2 The left figure shows a schematic diagram of a columnar grid, and the right figure shows a schematic diagram of a voxel grid.

[0060] See also Figure 2First, for each column grid v i Calculate the arithmetic mean p of the geometric coordinates of all point cloud points within avg , calculate the distance from each point in the grid to p avg The distance is recorded as Calculate the coordinate center p of each column grid in the x and y directions c , calculate the distance from each point in the grid to p c The offset of After processing, each point cloud point in the grid is represented as All columnar grid features are represented by tensors, which are (D, P, N), where D is the vector dimension of each point cloud point after processing, P is the number of all non-empty columnar grids, and N is the number of point cloud points in each columnar grid. In this embodiment: the number of input vectors of point cloud points is 9, so D = 9; the number of P ≤ n, because the columnar grid v i The inner point cloud points may be empty, and the empty columnar grid is not counted as a columnar grid feature; the point cloud points {p1, p2, …, p i ,…,p m}, so N is m.

[0061] Then, PointNet (a deep learning method for feature learning of point clouds) is used to fuse each point in the grid to generate a feature tensor of size (C, P, N), where C is the number of feature channels of each point cloud point in the grid after PointNet processing. This feature is a high-dimensional feature. Generally, the number of feature channels C of each point cloud point in the grid after PointNet processing is a multiple of 8. In this embodiment, C is 64.

[0062] Finally, maximum pooling is used for each columnar grid, that is, the maximum eigenvalue of all points in the columnar grid is retained and the maximum eigenvalue is represented as the corresponding columnar grid feature to obtain the columnar grid feature representation (C, P, 1), where P is the number of all non-empty columnar grids, and 1 represents the point cloud point with the maximum eigenvalue among all N point cloud points. The columnar grid feature representation after maximum pooling can also be simplified to (C, P).

[0063] See also Figure 2 For each voxel grid v i The arithmetic mean of the geometric coordinates and attribute information of all point cloud points in the avg ,y avg ,z avg ,reflectance avg , and is also expressed as v iThe processed voxel grid feature is represented by a tensor (C, Q), where C is the feature vector of each voxel grid after processing, and Q represents all non-empty voxel grids. In this embodiment, the number of feature vectors of each voxel grid is 4, so C = 4. The number of Q ≤ n, because the voxel grid v i The inner point cloud points may be empty, and the empty voxel grid is not counted as a columnar grid feature.

[0064] It should be noted that the above steps still process the fusion features in the three-dimensional point cloud grid, and the following steps convert the three-dimensional point cloud grid into a bird's-eye view representation in two-dimensional space.

[0065] S12. Use a neural network to fuse the features between the three-dimensional point cloud grids in the set V, and transform the three-dimensional point cloud grid features into a two-dimensional bird's-eye-view representation (BEV) feature map F BEV The schematic diagram is as follows Figure 3 As shown:

[0066] Figure 3 The left figure shows a schematic diagram of converting a columnar grid into a bird's-eye view representation of a two-dimensional space, and the right figure shows a schematic diagram of converting a voxel grid into a bird's-eye view representation of a two-dimensional space.

[0067] In this embodiment, two-dimensional convolution is used to fuse the inter-grid features for the columnar three-dimensional grid features. First, the columnar grid is flattened to a two-dimensional feature plane grid along the height direction, and then the two-dimensional convolution is used to fuse the features between the two-dimensional feature plane grids, and finally a bird's-eye view representation of the two-dimensional space is formed. The following is a specific example:

[0068] For the columnar grid feature representation (C, P), each columnar grid P is projected along the height direction onto a two-dimensional grid on the two-dimensional feature plane, and the dimension of the feature channel of the two-dimensional grid is C. The projected two-dimensional plane is processed using the backbone network two-dimensional convolutional neural network of the PointPillar model to form a bird's-eye view representation of the two-dimensional space, represented as (D, H, W). D represents the number of channels of the bird's-eye view representation of the two-dimensional space, and H and W represent the size of the bird's-eye view representation of the two-dimensional space, respectively.

[0069] To further illustrate the formation process of the bird's-eye view representation of two-dimensional space, the following example is given:

[0070] Assume that the value ranges of the point cloud on the x, y, and z axes are [-51.2, 51.2], [-51.2, 51.2], and [-5, 3], respectively, and the columnar grid size is [0.2, 0.2, 8], that is, 102.4 / 0.2 = 512 grids can be formed in the x-axis direction, and 102.4 / 0.2 = 512 grids can be formed in the y-axis direction. Since the columnar grid height is 8, it can completely cover the z-axis direction, so there is only 8 / 8 = 1 columnar grid in the z-axis direction. Finally, a 512*512*1 columnar grid is formed.

[0071] Project the 512*512*1 columnar grids along the z-axis direction to obtain a 512*512 two-dimensional grid feature. The number of feature channels C of each two-dimensional grid is also 64. The projected two-dimensional grid feature is recorded as (64, 512, 512).

[0072] It should be noted that the 512*512*1 columnar grids include empty and non-empty columnar grids, and the two-dimensional grid feature value of the projection of the empty columnar grid is 0.

[0073] Then, the projected two-dimensional grid features are processed using the backbone network of the PointPillar model, and finally a bird’s-eye view representation F of the two-dimensional space is formed. BEV , expressed as (D, H, W), D represents the number of channels after being processed by the backbone network 2D convolutional neural network of the PointPillar model, H and W represent the size of the 2D bird's-eye view feature, which is the size of the above-mentioned 2D grid feature 512*512. The processing process of the backbone network 2D convolutional neural network of the PointPillar model is a common technology for existing point cloud target detection tasks.

[0074] In this embodiment, three-dimensional convolution is used to fuse inter-grid features for voxel grid features, and at the same time, the three-dimensional features in the Z-axis height direction in the three-dimensional space are continuously compressed, and finally the three-dimensional features are flattened into a bird's-eye view representation form in the two-dimensional space. The following is a specific example.

[0075] For the voxel grid feature representation (C, Q), according to the processing method of the SECOND model, the voxel grid feature is converted into a sparse tensor representation {C Q ,F Q}, where C Q represents the set of coordinate positions of the voxel grid, F QRepresents a collection of feature vectors of voxel grids. The sparse tensor representation is input into the three-dimensional sparse convolution backbone network of the SECOND model, and the three-dimensional features in the Z-axis height direction are continuously downsampled to obtain a three-dimensional feature representation that fully integrates the voxel grids, with a size of (C, Z, H, W). Then the channel dimension is reorganized into the Z dimension to form a two-dimensional bird's-eye view representation of (D, H, W). D = C*Z represents the number of channels of the two-dimensional bird's-eye view representation, and H and W represent the size of the two-dimensional bird's-eye view representation.

[0076] To further illustrate the formation process of the bird's-eye view representation of two-dimensional space, the following example is given:

[0077] Assume that the value ranges of the point cloud on the x, y, and z axes are [-54.0, 54.0], [-54.0, 54.0], and [-5, 3], respectively, and the columnar grid size is [0.075, 0.075, 0.2], that is, 108.0 / 0.075 = 1440 grids can be formed on the x-axis, 108.0 / 0.075 = 1440 grids can be formed on the y-axis, and 8 / 0.2 = 40 grids can be formed on the z-axis. Finally, a 1440*1440*40 voxel grid is formed.

[0078] It should be noted that the 1440*1440*40 voxel grid contains empty and non-empty voxel grids. The sparse tensor representation only operates on non-empty voxel grids. The number of feature channels of non-empty voxel grids is 4, and the feature value of empty voxel grids is 0. After three consecutive downsamplings along the x, y, and z directions (that is, the spatial scale is reduced to half of the original, rounded), and the inter-voxel features are extracted, a three-dimensional feature of size 180*180*5 is formed. The three-dimensional feature is then downsampled in the z-axis direction to form a three-dimensional feature of size 180*180*2, and the number of three-dimensional feature channels is 128. This three-dimensional feature is a high-dimensional feature extracted by the three-dimensional sparse convolution backbone network of the SECOND model. The three-dimensional feature is denoted as (C, Z, H, W), which is (128, 2, 180, 180). The three-dimensional feature channels are reorganized into the Z dimension to finally form a bird's-eye view representation F of the two-dimensional space. BEV , expressed as (D, H, W), D = C * Z represents the number of channels of the two-dimensional bird's-eye view feature, which is 128 * 2 = 256, H, W respectively represent the size of the two-dimensional bird's-eye view feature, which is the size of the above-mentioned three-dimensional feature on the x-axis and y-axis 180 * 180. That is, (D, H, W) is (256, 180, 180). The three-dimensional sparse convolution backbone network processing process of the SECOND model of this embodiment is a common technology for existing point cloud target detection tasks.

[0079] It should be noted that the method of converting a three-dimensional grid into a two-dimensional representation form to form a bird's-eye view representation of a two-dimensional space is not limited to the above-listed embodiments.

[0080] S13. Encode the bird's-eye view representation of the two-dimensional space and form a code stream.

[0081] In order to remove the bird's-eye view representation F BEV Redundant information of the two-dimensional space needs to be further characterized by a bird's-eye view F BEV The present embodiment provides three embodiments, using the image coding method to encode and process the bird's-eye view representation F of the two-dimensional space. BEV One embodiment is to use a traditional image coding method to encode and process the bird's-eye view representation F of the two-dimensional space. BEV The other two embodiments are to encode and process the bird's-eye view representation F of the two-dimensional space based on the image encoding method of the neural network. BEV The above-mentioned image encoding method may include encoding of a single image or a sequence of images.

[0082] It should be noted that the code stream includes at least a bird's-eye view representation of the two-dimensional space F BEV information, and can also include a bird's-eye view representation of the two-dimensional space F BEV Additional information such as the coupling relationship with the machine intelligence task network. The coupling relationship refers to the bird's-eye view representation of the two-dimensional space F BEV Positional relation information in machine intelligence task networks.

[0083] Embodiment 11

[0084] The following is a bird's-eye view representation of the two-dimensional space encoded using the traditional image encoding method F BEV A specific example of forming a code stream:

[0085] First, the bird's-eye view representation F of the two-dimensional space BEV ∈D×H×W, transformed into a two-dimensional single-channel image or a 2D three-channel image Then convert the converted two-dimensional single-channel image or two-dimensional three-channel image I BEV Input into the existing video encoder under JPEG\HEVC\VVC and other standards and directly encode it into a bitstream.

[0086] Among them, converting the bird's-eye view representation of the two-dimensional space into a two-dimensional single-channel image is to convert the original two-dimensional bird's-eye view representation F BEV ∈D×H×W is divided into D parts along the channel direction, and each single 2D bird's-eye view channel feature 1×H×W is tiled from top to bottom and from left to right in the order of raster scanning into a large 2D single channel image In this embodiment, an integer is taken.

[0087] Among them, converting the bird's-eye view representation of the two-dimensional space into a two-dimensional three-channel image is to convert the original two-dimensional bird's-eye view representation F BEV ∈D×H×W is divided into D / 3 parts along the channel direction, and the two-dimensional bird's-eye view features of each three-channel 3×H×W are tiled from top to bottom and from left to right in the order of raster scanning to form a large-size two-dimensional three-channel image. In this embodiment, an integer is taken.

[0088] Example 12

[0089] This embodiment provides an image coding method based on a neural network to encode and process a bird's-eye view representation F of a two-dimensional space. BEV Form a code stream.

[0090] It includes two steps: downscaling the bird's-eye view representation of the two-dimensional space to obtain the downscaled bird's-eye view representation F′ of the two-dimensional space BEV ; Bird's-eye view representation of two-dimensional spatial downscaling F′ BEV Perform entropy coding to form a bit stream.

[0091] 1. Bird’s-eye view representation of two-dimensional space BEV The downscaling transformation obtains the bird's-eye view representation F′ of the two-dimensional space downscaling BEV This example lists two methods to obtain F′ BEV .

[0092] The following is the first bird's-eye view representation F′ obtained by downscaling in two-dimensional space: BEV Method. Bird's-eye view representation of two-dimensional space F BEV Downscaling transformation, including for F BEV The sparse characteristics of F are further integrated and compressed BEV Multi-scale sparse features in two-dimensional space and bird’s-eye view representation downsampling in two-dimensional space. Figure 4 As shown, the input is a bird’s-eye view representation F of the two-dimensional space BEV The output is a bird's-eye view representation F′ of a fused and compressed multi-scale sparse two-dimensional space BEV The bird's-eye view representation of the input two-dimensional space is simultaneously extracted through the local feature branch and the global feature branch to extract local and global compression features respectively, and then the two extracted features are spliced ​​along the channel, and the feature channel is recombined using the linear layer, and finally added to the original input branch to obtain the output multi-scale fusion compressed two-dimensional space bird's-eye view representation. Fig. 9 As shown in the figure, the input is a bird's-eye view representation of a two-dimensional space compressed by multi-scale fusion. After convolution downsampling, the output is a bird's-eye view representation F′ of a two-dimensional space downscale. BEV The following is a detailed description of the steps:

[0093] (1) Extract local compression features through the local feature branch. The local feature branch of this embodiment uses a residual convolution module to extract local compression features. The residual convolution module uses a local small convolution operation with a K×K convolution kernel. The actual receptive field is K×K, and the bird's-eye view representation feature dimension of the input two-dimensional space is D×H×W. After passing through the residual convolution module, a local compression feature of the same size of D×H×W is generated. In order to maintain appropriate computational complexity, this embodiment uses a convolution size of K=3.

[0094] (2) Extracting global compression features through the global feature branch. The global feature branch of this embodiment uses an enhanced ConvNeXt module to extract global compression features. Figure 4 The specific implementation process is as follows Figure 5 As shown in the figure, a channel-by-channel large convolution with a convolution kernel size of K×K is used, and a channel-by-channel dilated large convolution with a dilation rate of S and a convolution kernel size of K×K is added. After layer normalization (a normalization technique used in deep neural networks), 1×1 convolution, and then processing with the GELU neural network activation function and 1×1 convolution, the processed output is finally added to the input of the global feature branch to obtain the global compressed feature.

[0095] The actual receptive field of the enhanced ConvNeXt module of this embodiment is (1+(K-1)*S)×(1+(K-1)*S), and the input bird's-eye view representation feature dimension is D×H×W. After passing through the enhanced ConvNeXt module, a global feature of the same size of D×H×W is generated. A channel-by-channel dilated large convolution is added to the ConvNeXt module of the prior art to extract global compression features. By setting different K and S values, different convolution receptive ranges, i.e., different global receptive capabilities, can be achieved. In order to maintain appropriate computational complexity, K=7 and S=3 in this embodiment.

[0096] (3) Fusing the above local compression features and global compression features. In this embodiment, the fusion process is to splice the local compression features and the global compression features along the channel dimension. At this time, the feature dimension is 2D×H×W, and then reorganize them through the linear layer to obtain a feature dimension of D×H×W.

[0097] (4) The above fused features are added to the original input two-dimensional bird's-eye view representation to obtain the final output multi-scale fused and compressed two-dimensional bird's-eye view representation, thereby achieving the purpose of multi-scale feature fusion compression unique to the two-dimensional bird's-eye view representation.

[0098] (5) The bird's-eye view representation of the two-dimensional space compressed by the multi-scale fusion mentioned above is downsampled D*D times in the two-dimensional space scale through a convolution operation with a step size of D, and the output is the bird's-eye view representation F′ of the two-dimensional space downscaling BEVIn this embodiment, different D values ​​can be set to achieve downsampling at any scale, such as D=2.

[0099] The following is the second way to get the bird's-eye view representation F′ of the two-dimensional spatial downscaling: BEV Method. Bird's-eye view representation of two-dimensional space F BEV Downscaling transformation, including for F BEV The sparse characteristics of F further enhance the compression BEV Sparse features in two-dimensional space and downsampling of bird's-eye view representation in two-dimensional space. Figure 6 As shown, the input is a bird’s-eye view representation F of the two-dimensional space BEV , the output is a bird's-eye view representation of a multi-scale enhanced compressed two-dimensional space. The bird's-eye view representation of the input two-dimensional space is simultaneously extracted through the local feature branch and the global feature branch to extract the local and global compressed attention features respectively, and then the two extracted features are multiplied element by element and added to the original input branch to obtain the output multi-scale enhanced compressed two-dimensional space bird's-eye view representation. Fig. 9 As shown in the figure, the input is a bird's-eye view representation of a two-dimensional space with multi-scale enhancement compression. After convolution downsampling, the output is a bird's-eye view representation F′ of a two-dimensional space downscale. BEV The following is a detailed introduction to the implementation steps of the second method:

[0100] (1) Extract local compression features through the local feature branch. The local feature branch of this embodiment uses a residual convolution module to extract local compression features. The residual convolution module uses a local small convolution operation with a K×K convolution kernel. The actual receptive field is K×K. The bird's-eye view representation feature dimension of the input two-dimensional space is D×H×W. After passing through this module, local features of the same size of D×H×W are generated. In order to maintain appropriate computational complexity, this embodiment uses a convolution size of K=3.

[0101] (2) Extracting globally compressed attention features through the global feature branch. The global feature branch of this embodiment includes a visual attention module, a residual convolution module, a linear layer, and a sigmoid activation function. The visual attention network module is used to extract and output global compressed features, which are then converted into globally compressed attention features through a residual convolution module (the same as the residual convolution module processing process of the local feature branch in step (1) of this method), a linear layer, and a sigmoid activation function.

[0102] (3) Fusion of local compression features and global compression attention features. In this embodiment, the fusion process is specifically to multiply the global compression attention features of dimension D×H×W by the local compression features outputted in step (1) of this method element by element to obtain multi-scale attention features.

[0103] Among them, the visual attention network module includes layer normalization, 1×1 convolution, GELU neural network activation function, large core attention block and 1×1 convolution, such as Figure 7 As shown in the figure, the bird's-eye view representation of the input two-dimensional space is processed by layer normalization (a normalization technique used in deep neural networks), 1×1 convolution, GELU neural network activation function, large kernel attention block, and 1×1 convolution, and finally outputs the global compressed feature. The number of channels of the output global compressed feature is the number of input feature channels D.

[0104] Among them, Figure 8 As shown, the large-core attention block includes: K×K channel-by-channel convolution, expansion rate S, channel-by-channel expansion convolution with a convolution kernel size of K×K, and 1×1 convolution. The actual receptive field of the visual attention network module is (1+(K-1)*S)×(1+(K-1)*S), which is used to extract and output global compression features. By setting different K and S values, different convolution receptive ranges can be achieved. Different convolution receptive ranges have different global receptive capabilities. In order to maintain appropriate computational complexity, K=7 and S=3 in this embodiment.

[0105] (4) The multi-scale attention features obtained after fusion are added to the bird's-eye view representation of the original input two-dimensional space to obtain the final output multi-scale enhanced and compressed two-dimensional space bird's-eye view representation, thereby achieving the purpose of enhancing and compressing the sparse features unique to the bird's-eye view representation.

[0106] (5) The bird's-eye view representation of the above multi-scale enhanced compression in the two-dimensional space is downsampled D*D times in the two-dimensional space scale through a convolution operation with a step size of D, and the output is the bird's-eye view representation F′ of the two-dimensional space downscaling BEV In this embodiment, different D values ​​can be set to achieve downsampling at any scale, such as D=2.

[0107] It should be noted that any of the above methods can be used to represent the bird's-eye view of the two-dimensional space F BEV Downscaled transformation into a two-dimensional downscaled bird's-eye view representation F′ BEV The process can be performed more than once, and any one of the methods can be selected for processing, or one or a combination of two methods can be selected for processing. In addition, it should be noted that the bird's-eye view representation F of the two-dimensional space BEV Downscaled transformation into a two-dimensional downscaled bird's-eye view representation F′ BEV The method is not limited to the above two methods, which are just listed embodiments.

[0108] 2. Downscaling bird’s-eye view representation of two-dimensional space F′ BEV Entropy coding compression is performed using an entropy model. The specific implementation is as follows Fig.10 As shown, the input is a downscaled bird's-eye view representation F′ in two-dimensional spaceBEV , after quantization and rounding, a downscaled bird's-eye view representation of the quantized two-dimensional space is formed Entropy model is used to estimate the downscaled bird's-eye view representation of the quantized two-dimensional space The probability distribution of , so the arithmetic coding model can represent the downscaling of the quantized two-dimensional space from a bird's eye view Encode to form code stream output.

[0109] Embodiment 13

[0110] This embodiment provides another image coding method based on a neural network to encode and process the bird's-eye view representation F of a two-dimensional space. BEV Method for forming a code stream. The method of this embodiment and the method of embodiment 12 "reduced scale bird's-eye view representation F' of two-dimensional space" BEV The process of "using entropy model to perform entropy coding compression" is the same, but the input is different. This embodiment 13 directly represents the bird's-eye view of the two-dimensional space F BEV Use entropy model for entropy coding compression. Fig.10 As shown, the input is a bird’s-eye view representation F of the two-dimensional space BEV , after quantization and rounding, a bird's-eye view representation of the quantized two-dimensional space is formed The entropy model is used to estimate the bird's-eye view representation of the quantized two-dimensional space. The probability distribution of , so the arithmetic coding model can represent the quantized two-dimensional space from a bird's eye view Encode to form code stream output.

[0111] Example 2

[0112] The present invention also provides a point cloud processing method embodiment, which encodes the bird's-eye view representation F of the two-dimensional space in embodiment 1. BEV The formed code stream is processed, such as Fig.11 As shown, the specific steps include:

[0113] S21, encoding and processing the bird's-eye view representation F of the two-dimensional space in Example 1 BEV The formed code stream is parsed to obtain a bird's-eye view representation of the reconstructed two-dimensional space.

[0114] S22, decoding the reconstructed two-dimensional bird's-eye view representation to obtain a two-dimensional bird's-eye view representation representing the three-dimensional point cloud grid information.

[0115] This embodiment provides three embodiments, using the image decoding method to parse the code stream and decode it to obtain a two-dimensional bird's-eye view representation of the three-dimensional point cloud grid information. One embodiment is to use a traditional image coding method to parse the code stream and decode it to obtain a two-dimensional bird's-eye view representation of the three-dimensional point cloud grid information. The other two embodiments are to parse the code stream and decode it to obtain a bird's-eye view representation of the two-dimensional space representing the three-dimensional point cloud grid information.

[0116] Embodiment 21

[0117] The following is a bird's-eye view representation of the two-dimensional space representing the three-dimensional point cloud grid information by parsing the code stream and decoding it using the traditional image coding method. Specific examples:

[0118] First, use the existing decoders under JPEG\HEVC\VVC and other standards to parse the bitstream and reconstruct it into a two-dimensional single-channel image. or a 2D three-channel image Then the two-dimensional single-channel image or the two-dimensional three-channel image is inversely transformed and decoded into a bird's-eye view representation F of the two-dimensional space. BEV ∈D×H×W.

[0119] The two-dimensional single-channel image is inversely transformed and decoded into a bird's-eye view representation of a two-dimensional space, which is to transform the two-dimensional single-channel image I BEV Divided into 1×H×W Two-dimensional single-channel images, and divide these The two-dimensional single-channel sub-images are spliced ​​in the channel dimension in a raster scanning order to form a bird's-eye view representation F of the two-dimensional space. BEV ∈D×H×W.

[0120] The two-dimensional three-channel image is inversely transformed and decoded into a bird's-eye view representation of a two-dimensional space, which is to transform the two-dimensional three-channel image I BEV Divided into 3×H×W A two-dimensional three-channel image, and these divided The two-dimensional three-channel sub-images are spliced ​​in the channel dimension in a raster scanning order to form a bird's-eye view representation of the two-dimensional space F BEV ∈D×H×W.

[0121] Embodiment 22

[0122] This embodiment also provides an image decoding method based on a neural network to parse the code stream and decode it to obtain a bird's-eye view representation of a two-dimensional space representing the three-dimensional point cloud grid information. It includes two steps: parsing the bitstream formed after encoding the bird's-eye view representation of the two-dimensional space to obtain a quantized two-dimensional downscaled bird's-eye view representation Downscaling Bird's-Eye Representation of Quantized Two-Dimensional Space Perform upscaling transformation to obtain a bird's-eye view representation of the two-dimensional space representing the three-dimensional point cloud raster information

[0123] 1. Parse the bitstream formed after encoding the bird's-eye view representation of the two-dimensional space to obtain a quantized bird's-eye view representation of the downscaled two-dimensional space

[0124] Use the entropy model to perform entropy decoding and reconstruction. The specific implementation is as follows Fig.10 As shown, the input bitstream, the entropy model is used to estimate the quantized two-dimensional spatial downscaled bird's-eye view representation The probability distribution of the arithmetic decoding model can parse and restore the downscaled bird's-eye view representation of the quantized two-dimensional space of the input bit stream according to the probability distribution

[0125] 2. Downscaling bird’s-eye view representation of quantized two-dimensional space Perform upscaling transformation to obtain a bird's-eye view representation of the two-dimensional space representing the three-dimensional point cloud raster information This example lists two methods to obtain

[0126] The following is the first upscaling transformation to obtain a bird's-eye view representation of the two-dimensional space representing the three-dimensional point cloud raster information: A bird's-eye view representation of quantized two-dimensional spatial downscaling Upscaling, including The sparse characteristics of the quantized two-dimensional space downscaled bird's-eye view representation are upsampled to obtain the upsampled bird's-eye view representation of the two-dimensional space and the upsampled bird's-eye view representation of the two-dimensional space is reconstructed by multi-scale fusion to obtain the two-dimensional bird's-eye view representation representing the three-dimensional point cloud grid information. like Fig.12 As shown, the input is a quantized two-dimensional spatial downscaled bird's-eye view representation After deconvolution upsampling, the output is a bird's-eye view representation of upsampling in two-dimensional space. Figure 4 As shown, the input is a bird's-eye view representation of the upsampled two-dimensional space, and the output is a two-dimensional bird's-eye view representation of the three-dimensional point cloud grid information. The input two-dimensional space upsampled bird's-eye view representation is simultaneously passed through the local feature branch and the global feature branch to extract local and global reconstruction features respectively, and then the two extracted features are spliced ​​along the channel, and the feature channel is recombined using the linear layer, and finally added to the original input branch to obtain the output two-dimensional space bird's-eye view representation representing the three-dimensional point cloud raster information. The following describes the method in detail:

[0127] (1) Bird’s-eye view representation of the above quantitative two-dimensional spatial downscaling The deconvolution operation with a step size of U is performed to upsample the image by U*U times in the two-dimensional space scale, and the output is a bird's-eye view representation of the upscaled two-dimensional space. In this embodiment, different U values ​​can be set to achieve arbitrary scale downsampling, such as U=2.

[0128] (2) Extract local reconstruction features through the local feature branch. The local feature branch of this embodiment uses a residual convolution module to extract local reconstruction features. The residual convolution module uses a local small convolution operation with a K×K convolution kernel. The actual receptive field is K×K. The dimension of the bird's-eye view representation feature of the upscaled input two-dimensional space is D×H×W. After passing through the residual convolution module, a local reconstruction feature of the same size of D×H×W is generated. In order to maintain appropriate computational complexity, this embodiment uses a convolution size of K=3.

[0129] (3) Extracting global reconstruction features through the global feature branch. The global feature branch of this embodiment uses an enhanced ConvNeXt module to extract global reconstruction features. Figure 4 The specific implementation process is as follows Figure 5 As shown in the figure, a channel-by-channel large convolution with a convolution kernel size of K×K is used, and a channel-by-channel dilated large convolution with a dilation rate of S and a convolution kernel size of K×K is added. After layer normalization (a normalization technique used in deep neural networks), 1×1 convolution, and then processing with a GELU neural network activation function and 1×1 convolution, the processed output is finally added to the input of the global feature branch to obtain the global reconstruction feature.

[0130] The actual receptive field of the enhanced ConvNeXt module is (1+(K-1)*S)×(1+(K-1)*S), and the dimension of the bird's-eye view representation of the upscaled input two-dimensional space is D×H×W. After passing through the enhanced ConvNeXt module, a global feature of the same size of D×H×W is generated. A channel-by-channel dilated large convolution is added to the ConvNeXt module of the prior art to extract global reconstruction features. Different convolution receptive ranges can be achieved by setting different K and S values, that is, different global receptive capabilities. In order to maintain appropriate computational complexity, K=7 and S=3 in this embodiment.

[0131] (4) Fusing the above local reconstruction features with the global reconstruction features. In this embodiment, the fusion process is to concatenate the local reconstruction features with the global reconstruction features along the channel dimension, where the feature dimension is 2D×H×W, and then reorganize them through a linear layer to obtain a feature dimension of D×H×W.

[0132] (5) Add the above features obtained by linear layer reorganization to the original input two-dimensional bird's-eye view representation to obtain the final output two-dimensional bird's-eye view representation representing the three-dimensional point cloud grid information The purpose of multi-scale feature fusion and reconstruction unique to the bird's-eye view representation of two-dimensional space is achieved.

[0133] The following is the second upscaling transformation to obtain a bird's-eye view representation of the two-dimensional space representing the three-dimensional point cloud raster information: A bird's-eye view representation of quantized two-dimensional spatial downscaling Upscaling, including The sparse characteristics of the quantized two-dimensional space downscaled bird's-eye view representation are upsampled to obtain a two-dimensional space upsampled bird's-eye view representation and multi-scale enhanced reconstruction is performed on the two-dimensional space upsampled bird's-eye view representation to obtain a two-dimensional space bird's-eye view representation representing the three-dimensional point cloud grid information. like Fig.12 As shown, the input is a quantized two-dimensional spatial downscaled bird's-eye view representation After deconvolution upsampling, the output is a bird's-eye view representation of upsampling in two-dimensional space. Figure 6 As shown, the input is a bird's-eye view representation of the upsampled two-dimensional space, and the output is a two-dimensional bird's-eye view representation of the three-dimensional point cloud grid information. Input At the same time, the local feature branch and the global feature branch extract local and global reconstruction features respectively, and then multiply the two extracted features element by element and add them to the original input branch to obtain the output of the two-dimensional space bird's-eye view representation of the three-dimensional point cloud grid information. The second upscaling transformation method is described in detail below:

[0134] (1) Bird’s-eye view representation of the above quantitative two-dimensional spatial downscaling The deconvolution operation with a step size of U is performed to upsample the image by U*U times in the two-dimensional space scale, and the output is a bird's-eye view representation of the upscaled two-dimensional space. In this embodiment, different U values ​​can be set to achieve arbitrary scale downsampling, such as U=2.

[0135] (2) Extract local reconstruction features through the local feature branch. The local feature branch of this embodiment uses a residual convolution module to extract local reconstruction features. The residual convolution module uses a local small convolution operation with a K×K convolution kernel. The actual receptive field is K×K. The bird's-eye view representation feature dimension sampled in the input two-dimensional space is D×H×W. After passing through this module, local features of the same size of D×H×W are generated. In order to maintain appropriate computational complexity, this embodiment uses a convolution size of K=3.

[0136] (3) Extracting the attention features of global reconstruction through the global feature branch. The global feature branch of this embodiment includes a visual attention module, a residual convolution module, a linear layer and a sigmoid activation function. The visual attention network module is used to extract and output global reconstruction features, and the global reconstruction features are then converted into global reconstruction attention features through a residual convolution module (the same as the residual convolution module processing process of the local feature branch in step (1) of this method), a linear layer and a sigmoid activation function.

[0137] (4) Fusion of local reconstruction features and global reconstruction attention features. In this embodiment, the fusion process is to multiply the global reconstruction attention features of dimension D×H×W by the local reconstruction features outputted in step (1) of this method element by element to obtain multi-scale attention features.

[0138] Among them, the visual attention network module includes layer normalization, 1×1 convolution, GELU neural network activation function, large core attention block and 1×1 convolution, such as Figure 7 As shown in the figure, the bird's-eye view representation of the input two-dimensional space upsampled is processed by layer normalization (a normalization technique used in deep neural networks), 1×1 convolution, GELU neural network activation function, large kernel attention block and 1×1 convolution, and finally outputs the global reconstruction feature. The number of channels of the output global reconstruction feature is the number of input feature channels D.

[0139] Among them, Figure 8 As shown, the large-core attention block includes: K×K channel-by-channel convolution, expansion rate S, channel-by-channel expansion convolution with a convolution kernel size of K×K, and 1×1 convolution. The actual receptive field of the visual attention network module is (1+(K-1)*S)×(1+(K-1)*S), which is used to extract and output global reconstruction features. By setting different K and S values, different convolution receptive ranges can be achieved. Different convolution receptive ranges have different global receptive capabilities. In order to maintain appropriate computational complexity, in this embodiment, K=7 and S=3.

[0140] (5) The multi-scale attention features obtained after fusion are added to the bird's-eye view representation of the original input two-dimensional space upsampled to obtain the final output of the two-dimensional space bird's-eye view representation representing the three-dimensional point cloud grid information. The purpose of enhancing the compression of sparse features unique to bird's-eye view representation is achieved.

[0141] It should be noted that any of the above methods can be used to represent the bird's-eye view of the two-dimensional space F BEV The process of upscaling to transform the bird's-eye view representation of the two-dimensional space upscaling can be performed more than once, and any one of the methods can be selected for processing, or one or a combination of two methods can be selected for processing. In addition, it should be noted that the quantized two-dimensional space downscaling bird's-eye view representation Upscaling transforms to obtain a bird's-eye view representation of the two-dimensional space representing the three-dimensional point cloud raster information The method is not limited to the above two methods, which are just listed embodiments.

[0142] Embodiment 23

[0143] This embodiment provides another video image encoding method based on a neural network to parse the code stream to obtain a two-dimensional space bird's-eye view representation representing the three-dimensional point cloud grid information. The method of this embodiment has the same process as the step of "parsing the bitstream formed after encoding and compressing the bird's-eye view representation of the two-dimensional space" in Embodiment 22, but the input is different. Embodiment 23 uses an entropy model to perform entropy decoding and reconstruction on the bitstream. Fig.10 As shown, the input bitstream, the entropy model is used to estimate the bird's-eye view representation of the quantized two-dimensional space The probability distribution of the arithmetic decoding model can parse the input code stream according to the probability distribution to obtain a bird's-eye view representation of the two-dimensional space representing the three-dimensional point cloud grid information. Output.

[0144] It should be noted that the bitstream includes at least the bird's-eye view representation F of the two-dimensional space of the coding compression BEV information, and can also include a bird's-eye view representation of the two-dimensional space F BEV Additional information such as the coupling relationship with the machine intelligence task network. The coupling relationship refers to the bird's-eye view representation of the two-dimensional space F BEV Positional relation information in machine intelligence task networks.

[0145] The following is a bird's-eye view representation of the two-dimensional space using the three-dimensional point cloud raster information And additional information to complete a specific example of machine vision task prediction based on bird's-eye view representation:

[0146] Based on the bird's-eye view representation of the two-dimensional space in the additional information F BEV The coupling relationship with the machine intelligence task network determines the sub-network segmentation position of the machine intelligence task network, and decodes and reconstructs the The input is then fed into the subnetwork of the machine intelligence task network to complete the machine vision task prediction.

[0147] In this embodiment, the SECOND target detection network is used as the machine intelligence task network. The additional information obtained by decoding determines that the sub-network of the SECOND network includes a region proposal network module and a target detection head module, wherein the region proposal network module further represents the reconstructed bird's-eye view. Extract detailed information and output a bird’s-eye view representation F of the two-dimensional space to be predicted RPN , a bird's-eye view representation of the two-dimensional space to be predicted F RPN The target detection task head outputs the 3D target box position information.

[0148] Among them, the regional proposal network diagram of the SECOND backbone network is as follows Fig.13 As shown, the input decoding reconstruction is A convolutional neural network is used to extract feature expressions of three different scales. The feature expressions of three different scales are represented as (C, H / 2, W / 2), (2C, H / 4, W / 4), and (4C, H / 8, W / 8). Finally, the features of the three different scales are upsampled to three feature expressions of the same scale through a deconvolutional neural network. The feature expressions of the same scale are represented as (2C, H, W), (2C, H, W), and (2C, H, W). The three feature expressions of the same scale are spliced ​​and fused in the channel direction to form a bird's-eye view representation F of the two-dimensional space to be predicted. RPN , recorded as (8C,H,W). Then, the bird's-eye view representation F of the two-dimensional space to be predicted RPN The 3D target frame position information is outputted into the target detection head module. The SECOND model is an existing technology and will not be described in detail here.

[0149] like Fig.14 As shown, this embodiment proposes a point cloud coding processing device based on a bird's-eye view representation of a two-dimensional space, comprising:

[0150] 1. A three-dimensional point cloud grid feature extraction module, which is used to extract features of point cloud points in a three-dimensional point cloud grid set V as grid features. Specifically, it includes:

[0151] The geometric and attribute information of the point cloud points in the 3D point cloud grid is processed and fused, and the fused features are represented as the feature information of the 3D grid.

[0152] 2. Bird's-eye view representation module of 3D point cloud raster transformation to 2D space

[0153] The three-dimensional point cloud grid conversion two-dimensional bird's-eye view representation module is used to use a neural network to fuse the features between the three-dimensional point cloud grids in the set V, and convert the three-dimensional point cloud grid features into a two-dimensional bird's-eye view representation F BEV The bird's-eye view characterization module for transforming the three-dimensional point cloud grid into a two-dimensional space adopts the following different characterization forms for the columnar grid and the voxel grid respectively:

[0154] 1) Columnar grids use two-dimensional convolution to fuse inter-grid features for columnar three-dimensional grid features. First, the columnar grid is flattened to a two-dimensional feature plane grid along the height direction, and then two-dimensional convolution is used to fuse the features between the two-dimensional feature plane grids, and finally a bird's-eye view representation of the two-dimensional space is formed;

[0155] 2) For voxel grid features, three-dimensional convolution is used to fuse inter-grid features. At the same time, the three-dimensional features in the Z-axis height direction in the three-dimensional space are continuously compressed, and finally the three-dimensional features are flattened into a bird's-eye view representation in the two-dimensional space.

[0156] 3. Bird’s-eye view representation encoding module of two-dimensional space

[0157] The two-dimensional space bird's-eye view representation encoding module is used to encode the two-dimensional space bird's-eye view representation F BEV The code stream is compressed and encoded to form a code stream. It should be noted that the code stream includes at least the bird's-eye view representation F of the two-dimensional space compressed by the code. BEV information, and can also include a bird's-eye view representation of the two-dimensional space F BEV Additional information such as the coupling relationship with the machine intelligence task network. The coupling relationship refers to the bird's-eye view representation of the two-dimensional space F BEV Positional relation information in machine intelligence task networks.

[0158] This embodiment provides three specific examples, using the image encoder to represent the bird's-eye view of the two-dimensional space F BEV The encoding and compression are performed to form a code stream. The encoding process of the three specific examples can be referred to in the aforementioned embodiment 11, embodiment 12, and embodiment 13, which will not be described in detail here.

[0159] like Fig.15 As shown, this embodiment proposes a point cloud decoding processing device based on a bird's-eye view representation of a two-dimensional space, comprising:

[0160] 1. Code stream analysis module

[0161] The bitstream parsing module is used to parse the bitstream to obtain a bird's-eye view representation of the reconstructed two-dimensional space.

[0162] 2. Two-dimensional bird's-eye view representation decoding and processing module

[0163] The two-dimensional bird's-eye view representation decoding and processing module is used to decode the reconstructed two-dimensional bird's-eye view representation to obtain the two-dimensional bird's-eye view representation representing the three-dimensional point cloud grid information.

[0164] It should be noted that the bit stream formed by encoding and compressing the bird's-eye view representation of the two-dimensional space is decoded, in addition to at least including the bird's-eye view representation of the two-dimensional space F BEV information, and can also include a bird's-eye view representation of the two-dimensional space F BEV Additional information such as the coupling relationship with the machine intelligence task network. The coupling relationship refers to the bird's-eye view representation of the two-dimensional space F BEV Positional relation information in machine intelligence task networks.

[0165] This embodiment provides three specific examples, using an image decoder to parse the code stream and decode it to obtain a bird's-eye view representation of a two-dimensional space representing three-dimensional point cloud grid information. The decoding process of the three specific examples can be found in the aforementioned Embodiment 21, Embodiment 22, and Embodiment 23, which will not be elaborated here.

[0166] An embodiment of the present invention further provides an electronic device, including:

[0167] Memory for storing computer programs;

[0168] The processor is used to execute the program stored in the memory to implement the steps of the above-mentioned point cloud processing method embodiment based on bird's-eye view representation.

[0169] For the specific implementation of each step of this method and related explanations, please refer to Figure 1-15 The method embodiment shown is not described in detail here.

[0170] The memory of the electronic device mentioned in this embodiment may include a random access memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory.

[0171] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0172] The embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, each step of the above-mentioned point cloud processing method embodiment based on bird's-eye view representation is implemented. For the specific implementation of each step of the method and related explanations, please refer to Figure 1-15 The method embodiment shown is not described in detail here.

[0173] It should be noted that the various embodiments in this specification are described in a related manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0174] In particular, for the device, electronic device, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0175] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A point cloud processing method, characterized in that The steps include: S11, inputting a three-dimensional point cloud grid set V, and extracting features of point cloud points in the three-dimensional point cloud grid as grid features; S12, using a neural network to fuse features between three-dimensional point cloud grids in set V, and converting the three-dimensional point cloud grid features into a bird's-eye view representation of a two-dimensional space; S13. Encode the bird's-eye view representation of the two-dimensional space to form a code stream.

2. The point cloud processing method according to claim 1, characterized in that: The step of extracting features of point cloud points in the grid as grid features from the three-dimensional point cloud grid comprises: for the three-dimensional point cloud grid v in the set V i , process and fuse 3D point cloud grid v i The geometric and attribute information of the inner point cloud points is used to represent the fused features as a three-dimensional point cloud grid v i feature information.

3. The point cloud processing method according to claim 1, characterized in that: The encoding process of the bird's-eye view representation of the two-dimensional space to form a code stream includes the following steps: 1) Using a neural network-based image coding method to downscale the bird's-eye view representation of the two-dimensional space to obtain a downscaled bird's-eye view representation of the two-dimensional space; 2) Perform entropy coding on the downscaled bird's-eye view representation of the two-dimensional space to form a bit stream.

4. The point cloud processing method according to claim 3, characterized in that: Use any one of the following three methods or a combination of more than one to downscale the two-dimensional bird's-eye view representation to obtain a two-dimensional spatial downscaled bird's-eye view representation: Method 1: (1) Extracting local compression features through local feature branches; (2) Extracting global compression features through the global feature branch; (3) Fusion of local compression features and global compression features; (4) Adding the fused features to the original input two-dimensional bird's-eye view representation to obtain the final output multi-scale fused compressed two-dimensional bird's-eye view representation; (5) Reducing the spatial size of the bird's-eye view representation of the two-dimensional space compressed by multi-scale fusion to obtain a downscaled bird's-eye view representation of the two-dimensional space. Method 2: (1) Extracting local compression features through local feature branches; (2) Extracting global compressed attention features through the global feature branch; (3) Fusion of local compression features and global compression attention features; (4) Adding the fused features to the original input two-dimensional bird's-eye view representation to obtain the final output multi-scale enhanced compressed two-dimensional bird's-eye view representation; (5) Reducing the spatial size of the bird's-eye view representation of the two-dimensional space compressed by multi-scale enhancement, and obtaining a downscaled bird's-eye view representation of the two-dimensional space.

5. A point cloud processing method, characterized in that include: S21, parsing the bitstream to obtain a bird's-eye view representation of the reconstructed two-dimensional space; S22, decoding the reconstructed bird's-eye view representation of the two-dimensional space to obtain a bird's-eye view representation of the two-dimensional space representing the three-dimensional point cloud grid information.

6. The point cloud processing method according to claim 5, characterized in that: The parsing of the code stream includes obtaining a quantized two-dimensional spatial downscaled bird's-eye view representation.

7. The point cloud processing method according to claim 5, characterized in that: The decoding process of the reconstructed two-dimensional bird's-eye view representation to obtain the two-dimensional bird's-eye view representation representing the three-dimensional point cloud raster information includes: using a neural network-based image decoding method to upscale the quantized two-dimensional space downscaled bird's-eye view representation to obtain the reconstructed two-dimensional space bird's-eye view representation.

8. The point cloud processing method according to claim 7, characterized in that: The reconstructed two-dimensional bird's-eye view representation obtained by upscaling the quantized two-dimensional spatial downscaled bird's-eye view representation is processed by any one of the following two methods or a combination of more than one of them: Method 1: (1) Expanding the spatial size of the quantized two-dimensional spatial downscaling bird's-eye view representation to obtain a two-dimensional spatial upscaling bird's-eye view representation; (2) Extracting local reconstruction features through local feature branches; (3) Extracting global reconstruction features through the global feature branch; (4) Fusion of local reconstruction features and global reconstruction features; (5) adding the fused features to the quantized two-dimensional spatial downscaled bird's-eye view representation of the original input to obtain the reconstructed two-dimensional bird's-eye view representation of the final output; Method 2: (1) Expanding the spatial size of the quantized two-dimensional spatial downscaling bird's-eye view representation to obtain a two-dimensional spatial upscaling bird's-eye view representation; (2) Extracting local reconstruction features through local feature branches; (3) Extracting global reconstruction attention features through the global feature branch; (4) Fusion of local compression features and global reconstruction attention features; (5) The fused features are added to the quantized two-dimensional spatial downscaled bird's-eye view representation of the original input to obtain the reconstructed two-dimensional bird's-eye view representation of the final output.

9. A point cloud processing device, characterized in that include: The point cloud grid feature extraction module is used to extract the features of the point cloud points in the input three-dimensional point cloud grid as grid features; the three-dimensional point cloud grid transformation two-dimensional bird's-eye view representation module is used to use a neural network to fuse the features between the three-dimensional point cloud grids and transform the three-dimensional point cloud grid features into a two-dimensional bird's-eye view representation; The two-dimensional space bird's-eye view representation encoding module is used to encode the two-dimensional space bird's-eye view representation to form a code stream.

10. A point cloud processing device, characterized in that include: A code stream analysis module, used to analyze the code stream to obtain a bird's-eye view representation of the reconstructed two-dimensional space; The two-dimensional space bird's-eye view representation decoding module is used to decode the reconstructed two-dimensional space bird's-eye view representation to obtain the two-dimensional space bird's-eye view representation representing the three-dimensional point cloud grid information.

11. An electronic device, comprising: Memory for storing computer programs; A processor, configured to execute a program stored in a memory to implement the steps of the method described in any one of claims 1 to 4 and / or any one of claims 5 to 8.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 4 and / or any one of claims 5 to 8 are implemented.

Images