A three-dimensional point cloud scene understanding method and device for expanding a receptive field, and a medium

Abstract

A three-dimensional point cloud scene understanding method and device for expanding a receptive field, and a medium, constructs a linear large kernel convolution operator LinK module as a convolution kernel for computer vision perception, first obtains voxel data from a three-dimensional point cloud scene for feature extraction, a feature encoder is composed of four encoder layers, each encoder layer includes a one-time down-sampling layer, a sparse convolution module with a residual, and a LinK module, the down-sampled features are sent into the sparse convolution module with a residual and the LinK module in parallel, the features obtained by each encoder layer are added to obtain voxel features, which are used for downstream tasks of three-dimensional point cloud scene understanding, such as semantic segmentation or target detection, etc. The present application can be well integrated with existing methods, can be added to any backbone network based on sparse convolution for 3D scene understanding, realizes the expansion of the perception range in the three-dimensional point cloud scene understanding task while reducing the parameter quantity, and improves the performance of the network in multiple downstream tasks.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer software, and relates to three-dimensional point cloud scene understanding, in particular to a three-dimensional point cloud scene understanding method for expanding a receptive field, a device and a medium. BACKGROUND

[0002] In traditional 2D computer vision tasks, a larger receptive field can significantly improve the performance of models in downstream tasks such as semantic segmentation and object detection. Previously, some methods (RepLKNet, etc.) have explored the use of larger convolution kernels in the field of 2D image processing to obtain a larger range of perception, and have achieved great improvement in downstream tasks. Many studies have shown that a larger receptive field can bring more context information, thereby improving the performance of models in downstream tasks. At the same time, large convolution kernels can also help introduce more shape bias to the network, further improving the performance of segmentation, detection and other tasks.

[0003] Shape information is particularly important for this data modality of three-dimensional point clouds, so 3D networks can theoretically benefit more from the design of large convolution kernels. However, due to the sparsity of 3D data modalities, there are two obstacles to introducing large convolution kernels into 3D. On the one hand, there is an efficiency problem. If the size of the convolution kernel is simply expanded from 3x3x3 to 7x7x7, the model size will be 10 times larger, and the computational load and parameter quantity will also increase by the same proportion. On the other hand, there is an optimization problem. Because point clouds are sparsely distributed in space, if each position of the convolution kernel is assigned a certain weight, then a large number of idle positions will not participate in updating during network iteration, resulting in slow parameter updates. Previously, someone has tried to use an edge-sharing parameter method to alleviate these two problems and proposed a spatially shared weight convolution kernel (LargeKernel3D) with a size of 7x7x7. This design successfully achieved better performance on 3D semantic segmentation and object detection tasks, but the expansion of the receptive field is still limited. There are two defects in this design that prevent the further expansion of the convolution kernel: first, although the parameter quantity is controlled through weight sharing, the computational load still increases exponentially; second, the edge-sharing parameter design will cause a large amount of blur on larger convolution kernels, resulting in a decrease in network performance instead of an increase. SUMMARY

[0004] The problem to be solved by the present application is that existing convolution-based three-dimensional point cloud scene understanding methods usually use a convolution kernel with a size of 3x3x3, which limits the expansion of its effective receptive field, and there is room for further improvement in the performance of downstream tasks such as object detection and semantic segmentation. In particular, if the convolution kernel is simply expanded, the generation and calculation of three-dimensional convolution kernels will bring cubic-level time and space overheads.

[0005] The technical scheme of the present application is: a three-dimensional point cloud scene understanding method for expanding the receptive field, a linear large kernel convolutional operator LinK module is constructed as a convolution kernel for computer vision perception to realize the expansion of the perception range and the reduction of the parameter quantity in the three-dimensional point cloud scene understanding task, firstly, voxelization is performed on the three-dimensional point cloud scene to obtain voxel data, then 3D scene feature extraction is performed by a backbone network, the backbone network includes four layers of encoders, each layer of feature encoder includes a down-sampling layer, a sparse convolution module with residual error SRM and a LinK module, the down-sampled features are sent into the sparse convolution module with residual error SRM and the LinK module in parallel, the sparse convolution module with residual error SRM is stacked by two sparse basic modules SBB, and each sparse basic module SBB is realized by using the submanifold convolution of spconv; the features obtained by the multi-layer parallel sparse convolution module with residual error SRM and the LinK module are added to obtain voxel features, which are used for downstream tasks of three-dimensional point cloud scene understanding; wherein the LinK module includes linear generation of dynamic weights and block-based feature aggregation:

[0006] The LinK module dynamically weights the input features, uses global position encoding as a linear kernel generator, and is divided into two steps, firstly, dynamic weights are generated based on global position encoding: the absolute position of each voxel is linearly projected to have the same dimension as the feature dimension of the input LinK module, thereby generating the dynamic weight of the voxel at each position, then the feature and the generated weight are multiplied by the channel to obtain the dynamically weighted feature value, and then local features are aggregated to form a convolution-like operation;

[0007] Wherein the feature aggregation is based on blocks, the voxel space is divided into a plurality of non-overlapping blocks with a size of s x s x s, the feature aggregation operation is performed inside each block, and the features of all non-empty voxels in the block are aggregated, specifically: for all voxels in the input scene, the three-dimensional coordinates of the voxels in the space are taken, the corresponding hash values are calculated, voxels with the same hash value will be divided into the same block, and the features of non-empty voxels in the block are aggregated to obtain the features of voxels within the equivalent convolution kernel size s 3 as the output of the Link module.

[0008] Further, the structure of each layer of feature encoder is as follows:

[0009] out = ExtraConv (SparseResModule (F down ) + LinKModule (F down ))

[0010] Wherein out is the output of the feature encoder, F downdenotes the voxel feature after down-sampling, SparseResModule denotes the sparse convolution layer SRM, and ExtraConv is composed of 3x3x3 sparse convolution;

[0011] 1) For each LinK module, first define a weight generator as follows

[0012] w(x) = Φ(σ(x))

[0013] where σ(x) = Wx is a linear mapping function, Φ(*) is an activation function, and let all non-empty voxels be denoted by {p0, p1, …, p n}, and for any voxel, let its offset from the surrounding non-empty voxels be denoted by {δ0, δ1, δ2, …, δ m}, and the weight to be solved is Using the cosine function as the activation function, the following relationship holds:

[0014] w(δ i ) = w(p + δ i -p)

[0015] = cos(σ(p + δ i )) x cos(σ(p)) + sin(σ(p + δ i )) x sin(σ(p))

[0016] where i = 0, 1, 2, …, m, and p denotes the three-dimensional coordinates of the non-empty voxels, and thus the weight w(δ i ) corresponding to the offset δ i is obtained through linear mapping of the weight of the global coordinates and trigonometric function operation, and the local relative position relationship is represented by the weight of the global coordinates;

[0017] 2) In the convolution operation, let the convolution kernel coverage area be A = {p A +k | k ∈ Γ}, where p A is the center voxel coordinate of the area A, k is the offset relative to the center voxel, and the set of all offsets is denoted by Γ, denotes the feature of the voxel at p A +k, and then the local feature obtained by convolution at the position of p A is denoted as:

[0018]

[0019] As described in 1), in order to obtain the weight corresponding to the offset from the weight of the global coordinates, it is necessary to combine the trigonometric function, and thus two auxiliary features of the A region are introduced and​ As an intermediate representation:

[0020]

[0021]

[0022] And Since it is calculated based on global coordinates, it can be reused at all locations, and finally p is obtained using the following formula A The feature aggregation result at p

[0023]

[0024] Through auxiliary features And The final aggregation result is only related to the local offset, forming a convolution-like effect.

[0025] The block-based feature aggregation is specifically:

[0026] 1) Divide the voxel space into several non-overlapping blocks of size s x s x s, and each block is responsible for aggregating the features of all non-empty voxels in the block. The hash value of the three-dimensional coordinates of each voxel is calculated according to the absolute position of the voxel in space as follows:

[0027]

[0028] Where p (x) , p (y) and p (z) represent the three-dimensional coordinates of p position, s represents the side length of the block, voxels with the same hash value are classified into the same block, and the set of all blocks is denoted as:

[0029] B={B0,B1,…,B V}

[0030] 2) B j is covered as an overlapping area by the convolution kernel, j = 0, 1, …, V, and the global coordinates of voxel b are denoted as p b , and the feature at p b position is denoted as Auxiliary features calculated block by block:

[0031]

[0032]

[0033] Where and have aggregated the blocks B jThe voxel feature inside is equivalent to the effect of a convolution kernel with a size of s x s x s.

[0034] The application further provides an electronic device, including a storage medium and a processor, the storage medium is used for storing a computer program, the processor is used for executing the computer program, and the computer program is executed to realize the three-dimensional point cloud scene understanding method for expanding the receptive field.

[0035] The application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed to realize the three-dimensional point cloud scene understanding method for expanding the receptive field.

[0036] Based on the above analysis, the application proposes a new method to realize the class convolution large range perception scheme. The application makes two major improvements to the classic convolution operation. The first point is to replace the static kernel weight with a linear kernel generation module to provide weight only for the non-empty region because the 3D input is very sparse. At the same time, the module is shared in layers to avoid the case that some weights allocated to the blank region are not optimized in one iteration. The second point is to reuse the pre-computed aggregation results in non-overlapping blocks, which makes the computational complexity independent of the convolution kernel size. That is, the method proposed by the application can realize convolution kernels of any size with consistent overhead. Because the weight of the non-empty region is generated by the linear kernel, the application names the method as the linear large kernel class convolution operator, abbreviated as LinK. A large number of experiments conducted by the application on public benchmarks of 3D detection and semantic segmentation tasks prove the effectiveness of the scheme proposed by the application. In the 3D target detection task, the application uses the classic CenterPoint as the benchmark backbone network, adds the linear large kernel understanding module LinK designed by the application in the feature encoder of the CenterPoint, and maintains the rest unchanged. The application tests on the famous 3D autonomous driving dataset nuScenes (lidar track), and finally obtains NDS 73.4% on the test set, and obtains the first place in the ranking list, and the NDS index of the first place in the ranking list also exceeds 73 for the first time. In the semantic segmentation task, the application uses MinkowskiNet as the baseline network, and improves the performance (mIoU) by 2.7% in the SemanticKITTI test set.

[0037] Compared with the prior art, the application has the following advantages

[0038] The application utilizes the sparsity of 3D point cloud data and proposes an efficient large receptive field three-dimensional point cloud scene understanding model, which can more accurately capture the three-dimensional geometric information of objects compared with traditional methods, thereby being beneficial to the index improvement of many downstream tasks and achieving excellent performance in semantic segmentation and target detection tasks.

[0039] The present application first introduces a 21x21x21 super-large class convolution kernel in the field of 3D, combines a receptive field expansion strategy based on auxiliary features, and greatly improves the performance of the model on downstream tasks. Meanwhile, LinK has strong scalability and portability, and can be used as an efficient and powerful three-dimensional scene understanding method, and is applicable to any scene understanding model based on 3D convolution. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is the overall framework diagram of the model used by the present application.

[0041] Figure 2 is a process schematic diagram of the LinK feature extraction module proposed by the present application.

[0042] Figure 3 is a comparison schematic diagram of LinK and standard convolution operation proposed by the present application.

[0043] Figure 4 is an internal structure diagram of the LinK module proposed by the present application.

[0044] Figure 5 is a visualization result comparison diagram of the present application compared with the baseline network on the segmentation task, (a) is the visualization result of the baseline network, and (b) is the visualization result of the present application.

[0045] Figure 6 is a single-layer structure diagram of the feature encoder used by the embodiment of the present application.

[0046] Figure 7 is the overall flowchart of the method embodiment of the present application. DETAILED DESCRIPTION

[0047] The present application proposes an efficient large receptive field three-dimensional point cloud scene understanding model LinK, which utilizes the high sparsity of three-dimensional point cloud data, introduces a design of dynamically generating weights according to position information on the basis of sparse three-dimensional convolution, obtains local information through some auxiliary features that can be removed, and overall serves as a local feature extraction operator similar to convolution. The LinK operator of the present application includes three operation steps: first, the input feature is divided into several blocks according to the size of sxsxs; then, the absolute position coordinates of each voxel are linearly projected and used as dynamically generated weights; finally, the operator of the present application is applied within each block to perform feature aggregation operation. In order to further expand the effective receptive field, the present application introduces feature diffusion between blocks, so that the receptive field is expanded to (rxs) 3, which further improves the performance of the model on downstream tasks. The application respectively takes Minkowski net and CenterPoint as the baseline backbone network, and tests the effectiveness of the method on two tasks of semantic segmentation and target detection. Among them, the application ranks first in the nuScenes target detection public ranking list.

[0048] The application proposes an efficient large receptive field three-dimensional point cloud scene understanding method, which includes the following designs: (1) using the sparsity of point cloud data, the application designs a dynamic weight generation method instead of the original static convolution kernel, based on sparse convolution, so that the model only needs to calculate the weight for the non-empty position, and the non-empty position does not participate in the calculation, which greatly reduces the parameter amount while expanding the effective receptive field, as shown in Figure 3 (2) the aggregation results of adjacent blocks are repeatedly used, and the interaction between blocks is introduced to further increase the receptive field, reaching the size of the equivalent convolution kernel 21x21x21, which greatly improves the efficient receptive field.(3) The method of the application can be well integrated with existing models and can be added to any backbone network of 3D scene understanding based on sparse convolution to improve the performance of the network on multiple downstream tasks. The application tests on 3D semantic segmentation and target detection tasks, and the performance is significantly improved compared with the benchmark network. The application includes an input data preprocessing stage, a backbone network extracting 3D scene data feature stage, a downstream task decoding feature stage, a training stage and a testing stage. For the convenience of explaining the technical scheme of the application, the backbone network extracting 3D scene data feature stage is divided into several steps, such as steps 2) to 5) below, the downstream tasks of three-dimensional point cloud scene understanding include semantic segmentation task and target detection task, for different tasks and data sets, the application performs multi-frame fusion and voxelization operation to obtain voxel data of three-dimensional scene for feature extraction, Figure 1 is the system framework diagram used by the application, and the specific implementation steps are as follows.

[0049] 1) Data preprocessing stage: for different specific tasks and data sets, the preprocessing generally does not completely the same. Taking the nuScenes data set in target detection as an example, for the input point cloud sequence, the application respectively intercepts the points whose x / y coordinates fall within [-54m, 54m] and z coordinates fall within [-5m, 3m] according to the position distribution of the original point cloud, and then divides the point cloud into voxels according to the size of [0.075m, 0.075m, 0.2m] as the input of the 3D sparse convolution model. In the training stage, the range of random rotation, and random scaling, etc. In the actual use stage, for each frame of input point cloud, after the coordinate range is intercepted according to the above manner, no additional data enhancement is performed. The point cloud inputs for training and testing in the present application are denoted as

[0050] Taking the nuScenes dataset as an example, the specific process of data preprocessing is as follows:

[0051] 1. The point cloud sequence I originally sampled by nuScenes is as follows:

[0052] I = {S -10 ,…,S -2 ,S -1 ,P0,S1,S2,…,S 10 ,P1,…}

[0053] Where P0 and P1 represent key frames, the subscript numbers -10 to -1 of S represent the frame sequence to the left of the key frame P0, and the numbers 1 to 10 represent the frame sequence to the right of the key frame, the sampling rate of nuScenes is 20 FPS, and S contains about 1 s of context information.

[0054] 2. The point cloud sequence I' after multi-frame fusion of S according to a fixed number of frames is as follows:

[0055] I' = {P0,P1,P2,…}

[0056] Where P 0,1,2,… still represents a key frame, but it has fused the point cloud sequences of the previous 10 non-key frames, making the distribution of points more dense and the scene representation more detailed. I' is voxelized and subjected to a series of data enhancements to serve as the input of the model.

[0057] 2) Extraction of 3D scene features: a 3D sparse convolutional neural network is used as the basic network structure to perform frame-by-frame feature extraction on the input sequence generated in 1). Taking the target detection task as an example, the features of the input point cloud scene are extracted as M represents the features of the overhead view, W and L are the length and width of the scene in the overhead view, respectively, and C represents the number of channels. Specifically, the 3D sparse convolutional neural network serves as a feature encoder and is composed of four layers of down-sampling + sparse convolution modules. The structure of each layer is as shown in Figure 6 In each layer, first, the original features are down-sampled in space by a ratio of 1 / 2, and the size of the feature tensor is changed from [L, W, H] to Then, the down-sampled features are sent into the sparse residual module (SRM) and the LinK module in parallel. Each sparse convolutional network is stacked by two sparse basic blocks (SBBs), each of which is implemented by using the submanifold convolution of spconv to ensure the consistency of the sparse features. The specific structure of the LinK module will be described in detail in steps 3) and 4). After the multi-layer parallel sparse convolution and LinK module, the final down-sampled features are projected to the two-dimensional overhead view.

[0058] The specific calculation method of each layer of features in the encoder in step 2) is as follows:

[0059] 1. The input feature F in is down-sampled to reduce the size of the feature tensor by half in space:

[0060]

[0061] where N represents the batch size, L, W, and H represent the length, width, and height of the input feature in space, respectively, and C represents the feature dimension. In an embodiment, the down-sampling layer is specifically composed of a sparse convolution layer with kernel=3 and stride=2, a BatchNorm layer, and a ReLU layer.

[0062] 2. The down-sampled features are sent into the sparse convolution module and the LinK module of the present application in parallel:

[0063]

[0064]

[0065] where SparseResModule represents the sparse residual module, which is stacked by two sparse basic blocks (SBBs), each of which is composed of two sparse convolution layers with kernel=3, and a BatchNorm layer and a ReLU layer are inserted between the two sparse convolution layers. Finally, the residual term is added. The specific structure of the LinKModule will be described in detail in steps 3) and 4).

[0066] 3. The features after the sparse convolution and LinK module are added and projected to the two-dimensional plane, converting the originally distributed sparse three-dimensional features into dense two-dimensional overhead view features:

[0067] F bev =(F conv+F LinK ).view(N,CxH,L,W)

[0068] where view denotes the change of tensor shape, the output dimension of the feature is C, the spatial three-dimensional resolution is [L,W,H], and the dimension of the top-view feature tensor obtained by projecting onto the two-dimensional plane is When the application is applied to the target detection task, the output dimension of the feature is C=128, the spatial three-dimensional resolution is [180, 180, 2], and the dimension of the top-view feature tensor obtained by projecting onto the two-dimensional plane is

[0069] 3) Generation of linear kernel: the 3D sparse convolutional neural network of step 2) is composed of four layers of sparse convolutional modules with residual and LinK modules in parallel as a feature encoder. In the prior art method, the aggregation of local information and the expansion of the receptive field are usually coupled, resulting in the fact that each region is actually involved in repeated calculation several times. In the 3D point cloud scene, using traditional methods such as convolution to expand the receptive field will cause cubic-level growth of time and space overhead. At the same time, due to the sparsity of the spatial distribution of point clouds, if the convolution kernel is simply expanded, a large number of weights allocated to the blank will not participate in the calculation and will not participate in gradient update, not only wasting a large amount of time and space, but also making the convergence of the network a problem. Taking a single layer of 21x21x21 convolution as an example, if the input and output feature dimensions are 32 and 64 respectively, more than 18 million learnable parameters will be generated, but due to the sparsity of the spatial distribution of point clouds, most of them are idle during inference. The LinK module proposed in the application is a kind of three-dimensional point cloud scene understanding module that can efficiently expand the receptive field, by decoupling the feature aggregation and receptive field expansion operations, it can effectively realize the local feature extraction effect similar to large convolution kernel, while maintaining the linear computational complexity.

[0070] The application introduces the structure of parallel residual sparse convolution module and large kernel convolution LinK module in the feature extraction backbone network, as follows.

[0071] The structure of each layer of feature encoder is as follows:

[0072] out = ExtraConv(SparseResModule(F down )+LinKModule(F down ))

[0073] where F downwhere represents the down-sampled voxel feature, SparseResModule represents a 3D sparse convolution module with two layers of sparse basic blocks (SBB) and residual connections, LinKModule represents the proposed large kernel-like convolution module, and ExtraConv is composed of 3x3x3 sparse convolutions.

[0074] In the convolution operation, the region covered by the convolution kernel is denoted as A = {p A A where p A is the center voxel coordinate of the region A, k is the offset relative to the center voxel, and the set of all offsets is denoted as Γ, represents the feature of the voxel at p A The local feature obtained by convolution of the region at p is denoted as:

[0075]

[0076] Therefore, for two different coverage regions A1 and A2, the weights of the overlapping parts are generated with different local offsets, which cannot be directly reused, where reuse refers to the reuse of the weights of the overlapping parts, so as to realize a larger equivalent convolution kernel with smaller overhead and avoid repeated calculation.

[0077] In order to effectively reuse the weights of the overlapping parts of different regions, the present application uses global position encoding as a linear kernel generator. The generation of the linear kernel is divided into two steps. First, based on the global position encoding, the feature is weighted. The present application first linearly projects the absolute position of each voxel so that its dimension is the same as the dimension of the feature input into the LinK module. The absolute position is a three-dimensional coordinate in the voxel space, and the dynamic weight of the voxel at each position is generated to replace the static weight in the traditional convolution. Then, the feature input into the LinK module is multiplied by the generated weight by channel to obtain the dynamically weighted feature value. Specifically, the present application can use different mapping functions to realize the weighting operation by linear projection, such as exponential function, trigonometric function, linear function, etc. In the present method, the present application uses a weighting method based on trigonometric function. The feature is multiplied by the generated weight by channel to obtain the dynamically weighted feature value. Then, the auxiliary feature within the block is introduced to form a convolution-like operation. It is worth noting that in the process of local information aggregation, the auxiliary feature can be mathematically eliminated, as follows.

[0078] 1. For each LinK module, the present application first defines a weight generator as follows

[0079] w(x) = Φ(σ(x))

[0080] ​where σ(x) = Wx is a linear mapping function, Φ(*) is an activation function. Denote the three-dimensional coordinates of all non-empty voxels as {p0, p1, …, p n} and the offset of any voxel to its surrounding non-empty voxels as {δ0, δ1, δ2, …, δ m}, the weight to be solved is In this work, the cosine function is used as the activation function, i.e. Φ(*) = cos(*), so there is the following relationship:

[0081] w(δ i ) = w(p + δ i -p)

[0082] = cos(σ(p + δ i )) x cos(σ(p)) + sin(σ(p + δ i )) x sin(σ(p))

[0083] where i = 0, 1, 2, …, m, p represents the three-dimensional coordinates of non-empty voxels, and thus the local offset δ i The corresponding weight w(δ i ) can be obtained by linear mapping and trigonometric function operation of the weight of global coordinates (such as p and p + δ i ); thus, the present application can represent the local relative position relationship by the weight of global coordinates.

[0084] 2. Combine the cos and sin terms of the weight w(δ i ), and introduce two auxiliary features of the A region as intermediate representations, where p A is the center voxel coordinate of the region A, k is the offset relative to the center voxel, and the set of all offsets is denoted as Γ, represents the feature of the voxel at p A +k:

[0085]

[0086]

[0087] and Since it is calculated based on global coordinates, that is, based on global position encoding, it can be reused at all positions, that is, all continuous square regions segmented in the form of a sliding window can be used. Finally, the A region feature aggregation result is obtained using the following formula

[0088]

[0089] Aggregating features by computation assistance And The final aggregation result is only related to the local offset, and the mathematical effect is similar to convolution.

[0090] 4) Block-based feature aggregation:

[0091] The present application first divides the voxel space into several non-overlapping blocks of size s x s x s, and each block is responsible for aggregating the features of all non-empty voxels within the block. Specifically, for all voxels in the input scene, the present application takes their three-dimensional coordinates p (x) , p (y) and p (z) in space, calculates the corresponding hash value All voxels with the same hash value are divided into the same block. The present application denotes the set of all blocks as According to the formula in step 3), calculate and on the B j (j = 0, 1, 2, …, V) region respectively. They respectively obtain the features of voxels within the s 3 range with an equivalent convolution kernel size.

[0092] In order to further increase the large-scale perception ability of the model, the present application also integrates the features of adjacent blocks into the current block. Specifically, the present application searches for the features of adjacent non-empty blocks in the r x r x r range, and takes the arithmetic mean to obtain the feature fusion result in a larger range. Finally, each voxel can query the features of voxels within the (r x s) 3 range by adjacent block in a convolution-like manner.

[0093] The block-based feature aggregation is implemented as follows:

[0094] 1. Calculate the hash value of the three-dimensional coordinates of each voxel according to its absolute position in space as follows:

[0095]

[0096] Where p (x) , p (y) and p (z) represent the three-dimensional coordinates of the voxel, and s represents the block length. Voxel with the same hash value will be classified into the same block. The set of all blocks is denoted as

[0097] B = {B0, B1, …, B V}

[0098] 2. Calculate the auxiliary feature block by block as an intermediate representation:

[0099]

[0100]

[0101] The global coordinates of voxel b are denoted as p. b p b The features at the location are and Block B was gathered separately. j The voxel features within are equivalent to a convolution with a kernel size of s×s×s.

[0102] 3. To further increase the receptive field, query the features of adjacent blocks within a range r, and take the average to obtain the final features:

[0103]

[0104]

[0105] in To be with block B j Adjacent block sets (total r) 3 ( ), where j is the index of the block. Therefore, and Gathered into (r×s) respectively 3 Features of all voxels within the range. Finally, the feature g at each voxel p. p It can be represented as follows:

[0106]

[0107] Among them B j This indicates the block where voxel p is located. To be with block B j In (r×s) 3 The set of adjacent blocks within the range, for The number of non-empty voxels.

[0108] 5) Applications in downstream tasks: The efficient large receptive field point cloud scene understanding method LinK proposed in this invention implements 3D sparse convolution operations similar to large convolution kernels, while significantly reducing the complexity increase brought about by expanding the receptive field. Therefore, it can be inserted into any encoder based on a 3D sparse convolutional network to perform arbitrary downstream tasks. Experiments were conducted on two common tasks in 3D scenes: semantic segmentation and object detection.

[0109] For the semantic segmentation task, the present application uses MinkowskiNet as the baseline network. MinkowskiNet is composed of 4 layers of encoders and 4 layers of decoders, and the present application replaces the conventional convolution module in the encoder with a LinK-based convolution-like module. Specifically, the present application accesses a 4-layer upsampling module after the encoder as a feature decoder, gradually restores the feature scale to the original range, and finally generates a point-by-point segmentation result. The detailed structure of the network is shown in the (b) part of Figure 1 The multi-scale features output by the four stages of the encoder are respectively connected to the upsampling module, forming a classic U-Net structure. The present application selects the SemanticKITTI dataset to evaluate the effect of the model. It is a large-scale autonomous driving dataset, which contains more than 43,000 labeled frames, each frame contains more than 100,000 points, and a total of 20 object classes. The dataset contains a total of 22 driving sequences, of which the official sequence [00-07, 09-10] is the training set, sequence 08 is the validation set, and sequence [11-21] is the test set. The accuracy index is the average intersection over union mIoU (mean intersection over union).

[0110] For the object detection task, the present application uses CenterPoint as the baseline network. It is composed of four layers of encoders and a detection head. Further, for the single-stage detector used in the present method, the detection head is divided into a heat map head and a regression head. The heat map head predicts K-channel heat maps, where K is the number of target classes, and each class predicts a heat map. The ground truth of the heat map is obtained by projecting the center point of the target box to the bird's eye view, and then adding Gaussian blur. The regression head is mainly responsible for predicting the in-voxel offset the height of the object on the ground the three-dimensional size of the object and the yaw angle of the object (yaw), which is represented by the sine and cosine values of the angle The present application verifies the effect of the model on the nuScenes dataset commonly used in the field of 3D detection. It consists of 1000 labeled sequences, of which 700 sequences are the training set, and 150 sequences are the validation set and the test set, respectively. The dataset provides lidar, 6-view camera, and radar modal data for each scene, and in this work, only the lidar point cloud single modal data is used for performance evaluation. The evaluation index is NDS (nuScenes detection score), which not only evaluates mAP, but also evaluates the prediction errors of displacement, direction, speed, etc.

[0111] Specifically, for semantic segmentation task and object detection task, the backbone network proposed by the present application comprises 1 stem layer, 4 down-sampling stages, and then connects a segmentation head and a detection head, respectively. The stem layer and the down-sampling stages can be expressed as follows:

[0112] 1. The stem layer is composed of two layers of conv-batchnorm structure:

[0113]

[0114] wherein Conv represents a 3x3x3 sparse convolution layer, BN represents batchnorm, L, W, H represent length, width and height in space respectively, N is batch size, and c0 represents the number of output feature channels. In the embodiment, c0=32 is taken, that is, the stem layer increases the input feature dimension from 4 dimensions to 32 dimensions.

[0115] 2. The down-sampling stage is composed of one layer of down-sampling convolution layer with stride=2 and two sparse basic modules (SBB) and LinK modules connected in parallel:

[0116]

[0117]

[0118]

[0119]

[0120] wherein F conv and F LinK represent the outputs of the residual layer and the LinK module respectively, F out represents the final output of the stage, and c l represents the feature dimension of the lth layer.

[0121] 3. The segmentation head is composed of 4 upsampling stages. For each upsampling stage, let the input feature be F in , the output feature be F out_s , and the length, width and height of the input feature be L, W and H respectively, then the process can be expressed as:

[0122]

[0123]

[0124] 4. The detection head is divided into two parts, one part regresses the heat map distribution of the object center point, and the other part regresses other attributes of the detection box. They are represented as follows:

[0125]

[0126]

[0127] where F bev is the two-dimensional bird's eye view feature obtained by compressing the height dimension in step 2), CenterHead and RegHead are both two-layer convolutional layers, C represents the number of categories to be detected, and a represents some attribute of the detection box, including height, three-dimensional size, rotation angle, speed, etc.

[0128] 6) Training phase: for the initialization of the model, the networks in steps 2) to 5) use the default weight initialization of PyTorch, and the heat map head of the target detection branch in 5) follows the settings of CenterPoint, initialized to -2.19. For the semantic segmentation task, the loss function is the sum of the cross-entropy loss and the Lovasz loss. For the target detection task, the target center heat map is supervised using the focal loss to alleviate the class distribution imbalance problem; the regression head uses the L1 loss of each attribute, and the total loss is obtained by weighting the two according to a certain proportion. The total loss is optimized using the Adam optimizer, and the network parameters are updated through the back propagation algorithm. Steps 2) to 5) are repeatedly executed until the iteration number is reached.

[0129] The specific calculation process of the training loss function in step 6) is as follows:

[0130] L seg = CE (outputs, targets) + Lovasz (outputs, targets)

[0131] L det = L hm + λL loc

[0132] Where the loss function L seg of semantic segmentation is composed of cross-entropy loss (CE) and Lovasz loss. The loss function L det of target detection includes: the heat map loss L hm of the center point of the target detection box, and the L1 loss L loc of each attribute of the detection box, and the heat map loss is supervised using the focal loss. The loss mixing weight λ = 0.25.

[0133] 7) Test stage: Given the input point cloud scene. For the semantic segmentation task, the classification is predicted point by point (SemanticKITTI dataset contains 19 classes in total). For the object detection task, up to 500 candidate boxes and corresponding confidence scores are predicted for each scene, filtered by a confidence threshold t (t = 0.1 is selected in this work) and post-processed by NMS non-maximum suppression to obtain the final detection results.

[0134] The implementation of the present application is described below with an embodiment, as shown in the following table. Figure 7 The performance of semantic segmentation and object detection tasks is verified on the SemanticKITTI and nuScenes datasets respectively, and the Python 3 programming language and PyTorch 1.10.1 deep learning framework are used for implementation.

[0135] 1) Data preprocessing stage: For the object detection task, the sampling rate of the nuScenes dataset is 20FPS, but the data annotation is sparse, with 2 key frames annotated per second. For nuScenes, the present application takes the first 10 frames, i.e. 0.5s of point cloud scenes, based on the key frames with label information, and integrates the current key frame as the input of the model. For the semantic segmentation task, the SemanticKITTI used by the present application is a dense annotation dataset, annotated at 10FPS per second. In the training stage, for each frame of point cloud input, the order of all input points is randomly shuffled, random scaling with a ratio of 0.9-1.1 is performed, and random rotation within a range of is performed. Random jitter is performed on all points (random displacement with a standard deviation of 0.5 is performed point by point), and finally converted to Tensor format, batched and shuffled data loading order. In the test stage, no additional data augmentation is performed for each frame of point cloud input.

[0136] 2) Extraction of 3D scene features: 3D sparse convolutional neural network is used as the basic network structure to extract features from the input sequence generated in 1) frame by frame. The overall feature encoder structure is shown in Figure 1 (a), the input features are input into the sparse three-dimensional convolution module and the LinK module designed by the present application in parallel after being down-sampled, and the output results are added to obtain the final features. After four layers of encoder stacking, the output features are projected onto a two-dimensional plane to obtain the final features with a dimension of N x (C x H) x L x W. N is the batch size, C is the feature dimension, L, W and H correspond to the length, width and height of the spatial scale respectively. In the object detection task, the present application takes C = 128, [L, W, H] = [180, 180, 2]. After the 3D scene is encoded into dense features, it is connected to the detection head to obtain the object detection result. ​

[0137] 3) The generation process of the linear kernel is the core part of the LinK module, which decouples the operations of local feature aggregation and receptive field expansion, realizes the convolution-like operation of a larger equivalent convolution kernel, and introduces a larger receptive field while maintaining the linear computational complexity. In order to be able to reuse the weights at different positions and reduce repeated calculations in the sliding window process, the invention proposes to apply a linear mapping to the absolute position coordinates to dynamically generate the weights at each position. Specifically, first, the position dimension is aligned with the feature dimension using a linear mapping function, and then the global coordinates of the voxels are weighted using a trigonometric function to calculate the intermediate representation of each voxel feature and Implement an efficient convolution-like operation.

[0138] The internal structure of each LinK module is shown in Figure 4 First, 1x1x1 convolution is performed to realize the interaction between feature channels, and then the features are parallelly transmitted into the LinK module and the 3x3x3 convolution module, which preserves more detailed information and is more conducive to optimization using traditional sparse convolution. Unlike BatchNorm in traditional convolutional networks, the invention uses LayerNorm to enhance the interaction between channels.

[0139] 4) Block-based feature aggregation, as shown in Figure 2 First, the voxels are divided into blocks with a side length of s, Figure 2 3x3 blocks are divided, and then the intermediate representation of the features is calculated block by block according to the description in step 3) and Finally, the features of all voxels within the range of (rxs) 3 are averaged to eliminate the absolute position coordinates appearing in the intermediate representation, thereby achieving a result similar to the convolution operation with a size of (rxs) 3 Specifically, in the semantic segmentation task, the invention takes r=2 and s=3; in the object detection task, r=3 and s=7.

[0140] 5) In downstream tasks, the invention selects semantic segmentation and object detection, two common tasks in the autonomous driving scene, to evaluate the performance of the invention method. Specifically, the invention selects MinkowskiNet and CenterPoint as the baseline network, integrates the LinK module into the feature encoder, and keeps the rest unchanged. The overall structure is as follows Figure 1The evaluation is performed on the SemanticKITTI test set for the semantic segmentation task, and the model mIoU index of the LinK module is improved by 2.7 percentage points compared with the baseline network. For the target detection task, the evaluation is performed on the nuScenes dataset, and the NDS index of 73.4% is achieved on the test set, ranking first on the public leaderboard. In order to verify the expansion effect of the effective receptive field, the present application selects several scenes in the SemanticKITTI dataset for visualization, as shown in Figure 5 Figure 5 (a) is the visualization result of the baseline network (Baseline), Figure 5 (b) is the visualization result of the present application. In the figure, light color represents points predicted correctly, and dark color represents points predicted incorrectly. The method of the present application significantly improves the performance of semantic segmentation of large objects compared with the baseline network.

[0141] 6) Training phase, using cross-entropy loss and Lovasz loss to supervise semantic segmentation results. The focal loss of the heat map is used to supervise the center point position of the target class, and the L1 loss is used to supervise the position and size attributes of the detection box. Both are weighted and added according to 4:1, and the Adam optimizer is used to optimize the overall loss with a weight decay of 0.01. The total training round is 20 rounds, and the learning rate is scheduled using the one cycle strategy. The training is completed on 4 3090 GPUs, with 2 samples per GPU.

[0142] 7) Test phase, given the input point cloud scene. For the semantic segmentation task, the classification is predicted point by point (the SemanticKITTI dataset contains a total of 19 classes). For the target detection task, a maximum of 500 candidate boxes and corresponding confidence scores are predicted for each scene. After screening according to the confidence threshold t (t=0.1 is selected in this work) and NMS non-maximum suppression post-processing, the final detection result is obtained. In the evaluation stage, the average intersection over union (mIoU for short) index is used to evaluate the performance of the target detection model. On the SemanticKITTI dataset, MinkowskiNet is used as the baseline network, and the method of the present application improves the mIoU by 2.7 percentage points compared with the baseline network. On the nuScenes dataset, compared with the baseline network CenterPoint, the method of the present application improves by 3.1 percentage points on the validation set, and achieves NDS 73.4% on the test set, ranking first on the public leaderboard.​

Claims

1. A method for extended receptive field 3D point cloud scene understanding, characterized in that A linear large kernel convolution operator LinK module is constructed as a convolution kernel of computer vision perception to realize the expansion of the perception range and the reduction of the parameter amount in the three-dimensional point cloud scene understanding task. Firstly, a voxel data is obtained by voxelizing a three-dimensional point cloud scene, and then a backbone network is used for 3D scene feature extraction. The backbone network includes four layers of encoders. Each layer of feature encoder includes a down-sampling layer, a sparse convolution module with residual SRM and a LinK module. The down-sampled features are sent into the sparse convolution module with residual SRM and the LinK module in parallel. The sparse convolution module with residual SRM is stacked by two sparse basic modules SBB, and each sparse basic module SBB is implemented by using the sub-manifold convolution of spconv. The features obtained by the multi-layer parallel sparse convolution module with residual SRM and the LinK module are added to obtain voxel features, which are used for downstream tasks of three-dimensional point cloud scene understanding. The LinK module includes linear generation of dynamic weights and block-based feature aggregation: The LinK module dynamically weights the input features, uses global position encoding as a linear kernel generator, and is divided into two steps. Firstly, the dynamic weight is generated based on the global position encoding: the absolute position of each voxel is linearly projected to have the same dimension as the feature dimension of the input LinK module, thereby generating the dynamic weight of each position of the voxel. Then, the feature and the generated weight are multiplied by the channel to obtain the dynamic weighted feature value. Then, the local features are aggregated to form a convolution-like operation. wherein the feature aggregation is performed based on blocks, the voxel space is divided into a plurality of non-overlapping blocks with a size of sxsxs, and the aggregation operation of the features is performed within each block to aggregate the features of all non-empty voxels in the block. Specifically, for all voxels in the input scene, the three-dimensional coordinates of the voxels in the space are taken, and the corresponding hash values are calculated. The voxels with the same hash values will be divided into the same block, and the features of the non-empty voxels in the block are aggregated to obtain an equivalent convolution kernel with a size of s 3 The features of the voxels within the range are taken as the output of the Link module.

2. The method for understanding three-dimensional point cloud scenes by expanding the receptive field according to claim 1, characterized by a feature encoder. The specific implementation of each layer is as follows: 1) downsample the input features F in such that the size of their feature tensor is halved in space: Where N represents the batch size, L, W, and H represent the length, width, and height of the input feature in space, and C represents the feature dimension. 2) The down-sampled features are sent into the sparse convolution module with residual (SRM) and LinK module in parallel to get the corresponding outputs F conv and F LinK : Where SparseResModule represents a sparse convolution module with residual, which is stacked by two sparse basic modules SBB. Each sparse basic module SBB is composed of two layers of sparse convolution with a kernel size of 3x3x3, with BatchNorm and ReLU operations inserted in between. Finally, it is added with a residual term. LinKModule represents the LinK module. 3) The output features of the sparse convolution module SRM and the LinK module are added and projected onto a two-dimensional plane, converting the originally distributed sparse three-dimensional features into dense two-dimensional bird's eye view features: F bev = (F conv + F LinK ).view(N, C x H, L, W) wherein view denotes the change in the shape of the feature tensor, the output dimension of the feature is C, the spatial three-dimensional resolution is [L, W, H], and the dimension of the top-view feature tensor obtained by projecting onto a two-dimensional plane is 3. The three-dimensional point cloud scene understanding method for expanding the receptive field according to claim 1 or 2, wherein the structure of each layer of feature encoder is as follows: out = ExtraConv(SparseResModule(F down ) + LinKModule(F down )) where out is the output of the feature encoder, F down denotes the down-sampled voxel features, SparseResModule denotes a sparse convolution layer SRM, and ExtraConv is composed of a 3x3x3 sparse convolution. 1) For each LinK module, a weight generator is first defined as follows w(x)=Φ(σ(x)) where σ(x) = Wx is a linear mapping function, Φ(*) is an activation function, and let the three-dimensional coordinates of all non-empty voxels be {p0, p1, …, p n} For any voxel, let the offset of the voxel from the surrounding non-empty voxels be {δ0, δ1, δ2, …, δ m} The weight to be solved is Using the cosine function as the activation function, the following relationship holds: w(δ i ) = w(p + δ i -p) = cos(σ(p + δ i )) x cos(σ(p)) + sin(σ(p + δ i )) x sin(σ(p)) where i = 0, 1, 2, …, m, p represents the three-dimensional coordinates of the non-empty voxel, and thus, the offset δ i The corresponding weight w(δ i ) is obtained by linear mapping and trigonometric function operation of the weight of the global coordinates, and the relative position relationship of the local is represented by the weight of the global coordinates. 2) In the convolution operation, let the region covered by the convolution kernel be A = {p} A +k|k∈Γ}, where p A Let Γ be the coordinates of the center voxel of region A, and k be the offset relative to the center voxel. The set of all offsets is denoted as Γ. p A The characteristics of the voxel at +k, then p A Local features obtained by convolution in this region Represented as: As mentioned in 1), to get the weight corresponding to the offset from the weight of the global coordinate, the trigonometric function needs to be combined, thus introducing two auxiliary features of the A region and As intermediate representation: and Since it is calculated based on global coordinates, it can be reused at all locations, and finally p is obtained using the following formula A Feature aggregation result at By the auxiliary feature And The final resulting aggregation is only related to the local offset, resulting in a convolution-like effect.

4. The method of claim 3, wherein the method further comprises: The block-based feature aggregation is as follows: 1) The voxel space is divided into a plurality of non-overlapping blocks with a size of sxsxs, and each block is responsible for aggregating the features of all non-empty voxels in the block. The hash value of the three-dimensional coordinates of each voxel is calculated according to the absolute position of the voxel in space as follows: where p (x) , p (y) and p (z) represent the three-dimensional coordinates of the p position, s represents the side length of the block, voxels with the same hash value are classified into the same block, and the collection of all blocks is denoted as: B = {B0, B1,..., B V} 2) B j As the overlapping region covered by the convolution kernel, j = 0,1, …, V, record the global coordinates of the voxel b as p b , p b The feature at the position is Calculate the auxiliary feature block by block: wherein and respectively aggregate voxel features within block B j with an equivalent effect of a convolution with a kernel size of s x s x s.

5. The method of claim 4, wherein the method further comprises: To further increase the receptive field, the features of adjacent blocks are queried within a range r, and the average is taken to obtain the final feature: in To be with block B j In (r×s) 3 The set of adjacent blocks within the range, totaling r 3 , where j is the index of the block, therefore, and Gathered into (r×s) respectively 3 Features of all voxels within the range, ultimately, block B j The characteristic g of each voxel p within p It is expressed as follows: wherein is (r x s) 3 the number of non-empty voxels in the range.

6. The method of claim 1, wherein the method further comprises: The three-dimensional point cloud scene understanding task includes a semantic segmentation task and an object detection task. For different tasks and data sets, multi-frame fusion and voxelization operations are performed to obtain voxel data of a three-dimensional scene for feature extraction. A sparse convolutional neural network with a LinK module inserted is used as an encoder for feature extraction. For the semantic segmentation task, a 4-layer upsampling module is connected as a feature decoder after the encoder to gradually restore the feature scale to the original range, and finally generate a point-by-point segmentation result. For the object detection task, a detection head is connected after the encoder to realize prediction classification according to the features extracted by the encoder.

7. An electronic device, characterized by The computer readable storage medium stores a computer program, and the processor is configured to execute the computer program, and the computer program is executed to implement the three-dimensional point cloud scene understanding method with an enlarged receptive field according to any one of claims 1-6.

8. A computer readable storage medium, characterized by The computer readable storage medium stores a computer program, and the processor is configured to execute the computer program, and the computer program is executed to implement the three-dimensional point cloud scene understanding method with an enlarged receptive field according to any one of claims 1-6.

Images