Multi-scale sparse feature compression method and system based on three-dimensional sparse convolution

Abstract

The invention relates to a multi-scale sparse feature compression method and system based on three-dimensional sparse convolution, and the method comprises the steps: extracting multi-scale sparse features, fusing and compressing the multi-scale sparse features, providing a 3DED module, inputting 3D multi-scale sparse features, and outputting single-scale sparse features with two channels and a small space size; and single-scale sparse feature entropy coding and decoding: the input of the 3DED module is the compressed and coded code stream, and the output of the 3DED module is decoded 3D single-scale sparse features. And multi-scale sparse feature separation and expansion are carried out, a 3DSE module is provided, the input is decoded 3D single-scale sparse features, and the output is reconstructed 3D multi-scale sparse features. And target detection. According to the invention, compression and reconstruction of 3D multi-scale sparse features are realized. According to the method, the 3D machine vision task can be optimized under the condition of ensuring that the task performance is not obviously reduced, and a higher compression ratio is provided.

Description

Technical Field

[0001] The present invention relates to a feature encoding technology, and in particular to a multi-scale sparse feature compression method and system based on three-dimensional sparse convolution, belonging to the technical field of feature compression. Background Art

[0002] Traditional video coding methods are mainly optimized for the human visual system (HVS), focusing on the visual quality of the video. With the remarkable success of deep learning, the application of machine vision tasks has become increasingly widespread. However, it is undeniable that there are inherent differences between machine vision systems (MVS) and HVS. Therefore, applying traditional coding methods to machine vision tasks may not be the best choice. To address this problem, the MPEG organization launched a new generation of video codecs in 2019, called Video Coding for Machine Vision (VCM), which is specifically designed to provide compression coding for machine vision tasks. Since VCM is optimized for MVS, when applied to machine vision tasks, it can maintain task performance without significant degradation even when a very high compression ratio is achieved. Due to its superior performance in machine vision tasks compared to traditional methods, VCM has quickly gained widespread attention and attention, and many excellent VCM-related works have now appeared.In 2019, the University of Southern California proposed a new single-scale 2D feature compression method called BottleNet, which achieves feature compression by adding a bottleneck based on 2D convolution in the neural network (see AEEshratifar, A.Esmaili and M.Pedram, "BottleNet: A Deep Learning Architecture for Intelligent Mobile Cloud Computing Services," 2019IEEE / ACM International Symposium on Low Power Electronics and Design (ISLPED), Lausanne, Switzerland, 2019, pp. 1-6, doi: 10.1109 / ISLPED.2019.8824955.), and in 2020, Harbin Institute of Technology proposed a 2D multi-scale feature compression method called MSFC, which realizes the fusion and compression encoding of multi-scale features through ordinary 2D convolution, reducing the amount of transmitted data (see Z.Zhang, M.Wang, M.Ma, J.Li and X.Fan, "MSFC: Deep Feature Compression in Multi-Task Network,"2021IEEE International Conference on Multimedia and Expo(ICME),Shenzhen,China,2021,pp.1-6,doi:10.1109 / ICME51207.2021.9428258.), Kyung Hee University in South Korea proposed an end-to-end trainable multi-scale 2D feature compression method in 2023, which combines the two steps of multi-scale feature fusion and multi-scale feature compression, improving the task performance while providing better feature compression effect (reference Y.Kim et al., "End-to-End Learnable Multi-Scale Feature Compression forVCM,"in IEEE Transactions on Circuits and Systems for Video Technology,vol.34,no.5,pp.3156-3167,May 2024,doi:10.1109 / TCSVT.2023.3302858.).

[0003] However, the current VCM method can only be applied to 2D machine vision tasks, but not to 3D machine vision tasks. In recent years, with the rise of autonomous driving, 3D machine vision tasks based on LiDAR have also become a hot topic. Similar to the 2D field, there is also a compression-then-analysis (3D-CTA) architecture in the 3D field, such as Figure 1 As shown in the figure. In the 3D-CTA architecture, the mobile device first encodes the lidar point cloud into a bit stream using traditional point cloud coding methods, such as geometry-based point cloud compression (G-PCC) and video-based point cloud compression (V-PCC). Subsequently, the bit stream is transmitted to the cloud computing device through a wireless channel. After receiving the bit stream, the cloud computing device decodes it and analyzes the task results.

[0004] However, the existing 3D-CTA architecture has three defects: 1. The transmitted data is directly compressed by the original data collected by the sensor using the traditional point cloud compression method, which may lead to privacy leakage. 2. Traditional 3D point cloud coding is not efficient enough for machine vision tasks. In actual application scenarios, the number of mobile devices involved will be very large. The use of inefficient traditional 3D point cloud coding methods will put tremendous pressure on the transmission system and significantly increase maintenance costs. 3. When the number of mobile devices is large, cloud computing devices will generate more operations, which may lead to insufficient computing resources. Summary of the invention

[0005] In view of the shortcomings and gaps in the prior art, in order to make up for the defects of the existing VCM method that can only compress 2D features and the existing 3D-CTA method, the present invention proposes a multi-scale sparse feature compression method and system based on three-dimensional sparse convolution;

[0006] Specifically, in terms of machine vision task network segmentation, the present invention divides the neural network of the classic 3D object detection method VoxelR-CNN (reference Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wen gang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnn: Towards high performance voxel-based 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.) into two parts: a feature extractor network and a task network. The input of the feature extractor network is the point cloud to be detected, and the output is a 3D multi-scale sparse feature. The input of the task network is a 3D multi-scale sparse feature, and the output is the result of 3D object detection. The features to be compressed in the present invention are 3D multi-scale sparse features extracted by the feature extractor.

[0007] In terms of 3D multi-scale sparse feature fusion and compression, the present invention proposes a 3D multi-scale sparse feature fusion and compression (3DFC) module, whose input is 3D multi-scale sparse features, and its output is a single-scale sparse feature with 2 channels and a smaller spatial size.

[0008] In terms of single-scale sparse feature coding, the present invention proposes a 3D single-scale sparse feature entropy coding and decoding (3DEC) module, which can be divided into a 3D single-scale sparse feature entropy coding (3DEE) module and a 3D single-scale sparse feature entropy decoding (3DED) module. The input of the 3DEE module is the 3D single-scale sparse feature, and the output is the compressed encoded bit stream. The input of the 3DED module is the compressed encoded bit stream, and the output is the decoded 3D single-scale sparse feature.

[0009] In terms of multi-scale sparse feature separation and expansion, a 3D multi-scale sparse feature separation and expansion (3DSE) module is proposed. Its input is the decoded 3D single-scale sparse features, and its output is the reconstructed 3D multi-scale sparse features.

[0010] The present invention also proposes a multi-scale sparse feature compression system based on three-dimensional sparse convolution.

[0011] Terminology explanation:

[0012] 1. HVS (Human Visual System): human visual system.

[0013] 2. MVS (Machine Visual System): machine vision system.

[0014] 3. MPEG (Moving Pictures Experts Group): Moving Pictures Experts Group.

[0015] 4. VCM (Video Coding for Machine): Video coding for machine vision.

[0016] 5. LiDAR (Light Detection and Ranging): Laser detection and ranging system, also known as laser radar.

[0017] 6. 3D-CTA (3D-Compress Then Analyze): 3D compression then analysis architecture.

[0018] 7. G-PCC (Geometry Point Cloud Compression): A geometry-based point cloud compression method.

[0019] 8. V-PCC (Video Point Cloud Compression): A video-based point cloud compression method.

[0020] 9. 3DFC (3D multi-scale sparse features Fusion and Compression): 3D multi-scale sparse feature fusion and compression module, which fuses and compresses the input multi-scale sparse features to obtain the compressed and fused single-scale sparse features.

[0021] 10. 3DEC (3D single-scale sparse feature Entropy Codec): 3D single-scale sparse feature codec.

[0022] 11. 3DEE (3D single-scale sparse feature Entropy Encoder): 3D single-scale sparse feature entropy encoder.

[0023] 12. 3DED (3D single-scale sparse feature Entropy Decoder): 3D single-scale sparse feature decoder.

[0024] 13. 3DSE (3D multi-scale sparse features Separation and Expansion): 3D multi-scale sparse feature separation and expansion module, which separates and expands the input single-scale sparse features to obtain heavy single-scale sparse features.

[0025] The technical solution of the present invention is:

[0026] A multi-scale sparse feature compression method based on three-dimensional sparse convolution, comprising:

[0027] Multi-scale sparse feature extraction, including: Input the LiDAR point cloud P to be detected L After Voxel R-CNN front-end feature extraction, multi-scale sparse features F2, F3 and F4 are obtained;

[0028] The fusion and compression of multi-scale sparse features include: the sparse feature F2 with the largest size in the multi-scale sparse features is compressed in space to obtain a sparse feature F′2 with a space size equal to F3; F′2 is enhanced to obtain a sparse feature F″2 enhanced by F′2; F″2 and F3 are fused by 3DFC features to obtain a sparse feature F′2 fused by F3 23 ; F 23 After the spatial dimension is compressed, the sparse feature F′ with a spatial dimension equal to F4 is obtained. 23 ; F′ 23 After enhancement, we get F′ 23 Enhanced sparse features F″ 23 ; F″ 23 After 3DFC feature fusion with F4, we get F″ 23 The sparse feature F after fusion with F4 234 ; F 234 After enhancement, we get F 234 Enhanced sparse feature F′ 234 ; F′ 234 After channel compression, the output single-scale sparse feature F″ of 3DFC is obtained 234 ;

[0029] Single-scale sparse feature entropy encoding and decoding, including: 3DFC output single-scale sparse feature F″ 234 Sort by the same rules as G-PCC to obtain the sorted single-scale sparse features F″ s ; F″ s The geometric coordinates G are geometrically encoded by G-PCC to obtain the geometric coordinate code stream B1, F″ sThe characteristic attribute A is entropy encoded by DeepCABAC to obtain the characteristic attribute bitstream B2; the geometric coordinate bitstream B1 is geometrically decoded by G-PCC to obtain the decoded geometric coordinates The characteristic attribute bitstream B2 is decoded by DeepCABAC entropy to obtain the decoded characteristic attributes and After recombining, the single-scale sparse feature F at the decoding end is obtained dec ;

[0030] Multi-scale sparse feature separation and expansion, including: single-scale sparse feature F at the decoding end dec After three parallel channel upsampling operations, the channel upsampling features are obtained respectively and and After three feature attention modules, we get and After spatial upsampling and enhancement, a sparse feature with a spatial size equal to F3 is obtained. After spatial upsampling and enhancement again, the reconstructed sparse features of F2 are obtained. After spatial upsampling and enhancement, a sparse feature with a spatial size equal to F3 is obtained. and reconstruct sparse features After 3DSE fusion, the reconstructed sparse features of F3 are obtained After enhancement, we get Enhanced sparse features and reconstruct sparse features After 3DSE fusion, the reconstructed sparse features of F4 are obtained.

[0031] Object detection, including: reconstruction of multi-scale sparse features and After the 3D target detection algorithm, the final result of 3D target detection is obtained.

[0032] According to the preferred embodiment of the present invention, the LiDAR point cloud P is input L After the Voxel R-CNN front-end feature extraction network, multi-scale sparse features F2, F3 and F4 are obtained; including:

[0033] The input LiDAR point cloud P of Voxel R-CNN LIt is sent to the Voxel R-CNN front-end feature extraction network to obtain the output multi-scale sparse features F2, F3 and F4; among them, P L Expressed as Where N L represents the number of points in the LiDAR point cloud, 4 represents the four channels of X, Y, Z and Refc, Refc represents the reflectivity, and X, Y, Z represent the three coordinates of the point in the three-dimensional space; each multi-scale sparse feature in F2, F3 and F4 is expressed in the form of formula (I):

[0034]

[0035] In the formula, F represents 3D sparse features, N represents the number of feature points in the feature, (z i ,y i , x i , b i ) represents the geometric coordinates corresponding to the i-th feature point and the batch to which the i-th feature point belongs. M represents the dimension of the feature attribute of each feature point in the feature. The feature attribute corresponding to the i-th feature point represents the feature, (z max ,y max , x max ) represents the maximum possible value of the coordinates of each point in the sparse feature, which is the spatial size of the sparse feature.

[0036] Preferably, according to the present invention, the sparse feature F2 with the largest size in the multi-scale sparse features is compressed in space size to obtain a sparse feature F′2 with a space size equal to F3; comprising:

[0037] Input F2 into the spatial downsampling module to obtain the sparse feature F′2 with a spatial size equal to F3, as shown in formula (II):

[0038] F′2=Add(LReLU(Conv1(LReLU(Subm(F2)))), LReLU(Conv2(F2))) (II)

[0039] Where Subm(·) represents a 3D submanifold sparse convolution with a kernel size of 3 and an output channel number twice the number of input channels. Conv1(·) and Conv2(·) both represent ordinary 3D sparse convolutions with a kernel size of 3, a stride of 2, and an output channel number equal to the input channel number. LReLU(·) represents the LeakyReLU activation function. Add(·) represents a 3D sparse tensor addition operation, as shown in Equation (III) and Equation (IV):

[0040] G o =G m ∪G n(III)

[0041]

[0042] In the formula, G o [i] represents the output sparse feature F o The geometric coordinates corresponding to the i-th feature point; A o [i] represents the output sparse feature F o The characteristic attribute corresponding to the i-th feature point; G m [i] represents the first input sparse feature F m The geometric coordinates corresponding to the i-th feature point; A m [i] represents the first input sparse feature F m The characteristic attribute corresponding to the i-th feature point; G n [i] represents the second input sparse feature F n The geometric coordinates corresponding to the i-th feature point; A n [i] represents the first input sparse feature F n The feature attribute corresponding to the i-th feature point.

[0043] Preferably, according to the present invention, F″2 is subjected to an enhancement module to obtain a sparse feature F″2 enhanced by F′2, including: F′2 is aggregated and enhanced by two series-connected IRNBlocks to obtain the enhanced sparse feature F″2.

[0044] According to the preferred embodiment of the present invention, F″2 and F3 are subjected to a 3DFC feature fusion module to obtain a sparse feature F after fusion of F″2 and F3. 23 ,include:

[0045] The 3DFC feature fusion module is shown in formula (V):

[0046] F 23 =LReLU(Subm3(diffcat(LReLU(Subm2(LReLU(Subm1(F″2))))),F3)) (V)

[0047] In the formula, F 23 represents the sparse features after the fusion of F″2 and F3; Subm1(·), Subm2(·), and Subm3(·) all represent 3D submanifold sparse convolutions with a convolution kernel size of 3 and the number of output channels equal to the number of F3 channels; LReLU(·) represents the LeakyReLU activation function; diffcat(·) represents the connection method for 3D sparse features with different geometric coordinate information, as shown in formulas (VI) and (VII):

[0048] G c =G m∪G n (VI)

[0049]

[0050] In the formula, G c [i] represents the sparse feature F after output addition c The geometric coordinates corresponding to the i-th feature point; A c [i] represents the sparse feature F after output addition c The characteristic attribute corresponding to the i-th feature point; G m [i] represents the first input sparse feature F m The geometric coordinates corresponding to the i-th feature point; A m [i] represents the first input sparse feature F m The characteristic attribute corresponding to the i-th feature point; G n [i] represents the second input sparse feature F n The geometric coordinates corresponding to the i-th feature point; A n [i] represents the first input sparse feature F n The characteristic attribute corresponding to the i-th feature point; cat(·) represents a normal tensor connection operation.

[0051] According to the preferred embodiment of the present invention, F′ 234 After the channel downsampling module, the output single-scale sparse feature F″ of 3DFC is obtained 234 ,include:

[0052] F′ 234 After the channel downsampling module, the channel is downsampled by 32 times, and the number of channels is reduced from the original 64 to 2. The specific architecture of the channel downsampling module is expressed as formula (VIII):

[0053] F″ 234 =Comp(Comp(F′ 234 ,8),4) (VIII)

[0054] In the formula, F″ 234 is the output single-scale sparse feature of 3DFC; Comp(F in , N) represents the N-fold channel compression module, the number of channels of its output features is 1 / N of the number of input feature channels; Comp(F in , N) is represented by formula (IX), (X), (XI):

[0055] F inter1 =LReLU(Subm3(LReLU(Subm2(LReLU(Subm1(F in ))))) (IX)

[0056] F inter2 =LReLU(Subm4(F in ) (X)

[0057] F out =Add(F inter1 , F inter2 ) (XI)

[0058] In formula (IX), F in represents the input features of the channel compression module; F inter1 represents the first intermediate feature of the channel compression module; LReLU(·) represents the LeakyReLU activation function; Subm1(·) represents the submanifold sparse convolution with the number of output channels equal to the number of input channels 1 / N, and the convolution kernel size is 1; Subm2(·) represents the submanifold sparse convolution with the number of output channels equal to the number of input channels, and the convolution kernel size is 3; Subm3(·) represents the submanifold sparse convolution with the number of output channels equal to the number of input channels, and the convolution kernel size is 1; In formula (X), F inter2 represents the second intermediate feature of the channel compression module; Subm4(·) represents the submanifold sparse convolution whose output channel number is 1 / N of the input channel number and the convolution kernel size is 3; in formula (XI), F out represents the output features of the channel compression module, and Add(·) represents the 3D sparse tensor addition operation.

[0059] According to the preferred embodiment of the present invention, the output single-scale sparse feature F″ of 3DFC 234 Sort by the same rules as G-PCC to obtain the sorted single-scale sparse features F″ s ;include:

[0060] The output single-scale sparse feature F″ of 3DFC 234 The geometric coordinates G″ 234 Arrange the coordinates in the order of x from small to large, and then arrange the coordinates with the same x in the order of y from small to large. After arrangement, the coordinates with the same x and y are arranged in the order of z from small to large, and finally obtain the sorted geometric coordinates G; then according to the one-to-one correspondence between the original geometric coordinates and the feature attributes, 234 The feature attributes are rearranged to obtain the sorted feature attribute A; finally, the geometric coordinates G and the feature attribute A are recombined into the sorted single-scale sparse feature F″ s .

[0061] F″ s The geometric coordinates G are passed through the G-PCC geometric encoder to obtain the geometric coordinate code stream B1, F″ s The characteristic attribute A is passed through the DeepCABAC entropy encoder to obtain the characteristic attribute code stream B2; including:

[0062] According to the preferred embodiment of the present invention, the single-scale sparse feature F at the decoding end dec After three parallel channel upsampling operations, the channel upsampling features are obtained respectively and include:

[0063] F dec First, two serial channel upsampling modules are used to upsample the channel by 8 times and 4 times respectively. The number of channels changes from 2 to 64, as shown in formula (XII):

[0064]

[0065] In the formula, CExp(Fi n , N) represents an N-fold channel expansion module, and the number of channels of its output features is N times the number of channels of the input features; is the feature after upsampling of the first branch channel; among them, CExp(F in , N) is specifically represented by formula (XIII), (XIV), (XV):

[0066] F inter1 =LReLU(Subm3(LReLU(Subm2(LReLU(Subm1(F in ))))) (XIII)

[0067] F inter2 =LReLU(Subm4(F in ) (XIV)

[0068] F out =Add(F inter1 , F inter2 ) (XV)

[0069] In formula (XIII), F in represents the input features of the channel expansion module; F inter1 represents the first intermediate feature of the channel expansion module; LReLU(·) represents the LeakyReLU activation function; Subm1(·) represents the submanifold sparse convolution with the number of output channels being N times the number of input channels, and the convolution kernel size is 1; Subm2(·) represents the submanifold sparse convolution with the number of output channels being equal to the number of input channels, and the convolution kernel size is 3; Subm3(·) represents the submanifold sparse convolution with the number of output channels being equal to the number of input channels, and the convolution kernel size is 1; In formula (XIV), F inter2represents the second intermediate feature of the channel expansion module; Subm4(·) represents the submanifold sparse convolution with the number of output channels being N times the number of input channels and the convolution kernel size being 3; in formula (XV), F out represents the output feature of the channel expansion module, and Add(·) represents the 3D sparse tensor addition operation;

[0070] F dec Perform two channel upsampling at different magnifications to obtain the features after channel upsampling and As shown in formula (XVI) and formula (XVII):

[0071]

[0072] According to the preferred embodiment of the present invention, and After three feature attention modules respectively, we get and include:

[0073] Will and The inputs are respectively fed into three parallel independent feature point attention modules, and the features after attention perception are respectively output. and As shown in formula (XVIII), formula (XIX), and formula (XX):

[0074]

[0075] Where Att(·) represents the feature point attention module. The network structure of the feature point attention module is shown in formula (XXI), formula (XXII), formula (XXIII) and formula (XXIV):

[0076] F inter1 =Resblock(Resblock(F in )) (XXI)

[0077] F inter2 =Softmax(Subm1(Resblock(Resblock(F in )))) (XXII)

[0078] F out =Add(F in , F inter1 ×F inter2 ) (XXIII)

[0079] F res =Add(F inres,LReLU(Subm4(LReLU(Subm3(LReLU(Subm2(F inres ))))))) (XXIV)

[0080] In formula (XXI), F in represents the input features of the feature point attention module, F inter1 represents the first intermediate feature of the feature point attention module, Resblock(·) represents the 3D residual convolution block; the specific operation of Resblock(·) is shown in formula (XXIV), where F res represents the output features of the residual block, F inres represents the input features of the residual block, Subm2(·), Subm3(·), and Subm4(·) respectively represent the submanifold sparse convolution with the number of output channels being 1 / 2, 1 / 2, and 1 times the number of input channels, and the convolution kernel size is 3; in formula (XXII), F inter2 is the second intermediate feature of the feature point attention module, Subm1(·) represents the submanifold sparse convolution with an output channel of 1 and a convolution kernel size of 5, and Softmax(·) represents the Softmax function. In formula (XXIII), Add(·) represents the 3D sparse tensor addition operation.

[0081] According to the preferred embodiment of the present invention, After the spatial size upsampling module and the enhancement module, a sparse feature with a spatial size equal to F3 is obtained. include:

[0082] Input is a sparse feature of two IRNs and a spatial size upsampling module, and the output spatial size is equal to F3 As shown in formula (XXV) and formula (XXVI):

[0083]

[0084] F out =Add(LReLU(Conv1(LReLU(Subm(F in )))), LReLU(Conv2(F in )))) (XXVI)

[0085] Where IRN(·) represents the IRN module; Subm(·) represents a 3D submanifold sparse volume with a convolution kernel size of 3 and an output channel number twice the number of input channels; UPScale(·) represents the spatial size upsampling module, as shown in formula (XXVI), Fi n Represents the sparse features of the input; F outRepresents the output sparse features; Conv1(·) and Conv2(·) both represent transposed sparse convolution, the convolution kernel size is 3, the stride is 1, the number of output channels is equal to the number of input channels, and the size of the output sparse features is twice the size of the input sparse features; LReLU(·) represents the LeakyReLU activation function; Add(·) represents the 3D sparse tensor addition operation.

[0086] After the spatial size upsampling module and the enhancement module again, the reconstructed sparse features of F2 are obtained. include:

[0087] It is input again into two IRNs and a spatial size upsampling module, and the output spatial size is equal to the sparse features of F2 As shown in formula (XXVII):

[0088]

[0089] After spatial upsampling and enhancement modules, a sparse feature with a spatial size equal to F3 is obtained. include:

[0090] Input is a sparse feature of two IRNs and a spatial size upsampling module, and the output spatial size is equal to F3 As shown in formula (XXVIII):

[0091]

[0092] According to the preferred embodiment of the present invention, and reconstruct sparse features After the 3DSE fusion module, the reconstructed sparse features of F3 are obtained. include:

[0093] and Input into the 3DSE fusion module and output the reconstructed sparse features of F3 As shown in formula (XXIX):

[0094]

[0095] In the formula, represent and The fused sparse features, i.e. the reconstructed sparse features of F3 Subm1(·) and Subm2(·) both represent a convolution kernel size of 3, and the number of output channels is equal to 3D submanifold sparse convolution with the same number of channels; Conv1(·) represents 3D sparse convolution with a kernel size of 3, a stride of 2, and the number of output channels equal to the number of input channels; LReLU(·) represents the LeakyReLU activation function; diffcat(·) represents the connection method for 3D sparse features with different geometric coordinate information;

[0096] After the enhancement module, we get Enhanced sparse features include:

[0097] Input to two IRN modules connected in series, such as Figure 6 As shown, the output Enhanced sparse features As shown in formula (XXX):

[0098]

[0099] According to the preferred embodiment of the present invention, and reconstruct sparse features After the 3DSE fusion module, the reconstructed sparse features of F4 are obtained. include:

[0100] and Input into the 3DSE fusion module, output the reconstructed sparse features of F4 As shown in formula (XXXI):

[0101]

[0102] In the formula, represent and The fused sparse features, i.e. the reconstructed sparse features of F4 Subm1(·) and Subm2(·) both represent a convolution kernel size of 3 and an output channel number equal to The number of channels of the 3D submanifold sparse convolution is 3D; Conv1(·) represents a 3D sparse convolution with a kernel size of 3, a stride of 2, and the number of output channels equal to the number of input channels; LReLU(·) represents the LeakyReLU activation function; diffcat(·) represents the connection method for 3D sparse features with different geometric coordinate information.

[0103] According to the preferred embodiment of the present invention, multi-scale sparse features are reconstructed and After the back-end network of the 3D object detection algorithm VoxelR-CNN, the final result of 3D object detection is obtained; as shown in formula (XXXII):

[0104]

[0105] In the formula, R Det represents the final result of 3D object detection; TaskNetwork(·) represents the backend task network of Voxel R-CNN.

[0106] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a multi-scale sparse feature compression method based on three-dimensional sparse convolution when executing the computer program.

[0107] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a multi-scale sparse feature compression method based on three-dimensional sparse convolution.

[0108] A multi-scale sparse feature compression system based on three-dimensional sparse convolution, comprising:

[0109] The multi-scale sparse feature extraction module is configured as follows: Input the LiDAR point cloud P to be detected L After Voxel R-CNN front-end feature extraction, multi-scale sparse features F2, F3 and F4 are obtained;

[0110] The multi-scale sparse feature fusion and compression module is configured as follows: the sparse feature F2 with the largest size in the multi-scale sparse features is compressed in spatial size to obtain a sparse feature F′2 with a spatial size equal to F3; F′2 is enhanced to obtain a sparse feature F″2 enhanced by F′2; F″2 and F3 are fused by 3DFC features to obtain a sparse feature F′2 fused by F3. 23 ; F 23 After the spatial dimension is compressed, the sparse feature F′ with a spatial dimension equal to F4 is obtained. 23 ; F′ 23 After enhancement, we get F′ 23 Enhanced sparse features F″ 23 ; F″ 23 After 3DFC feature fusion with F4, we get F″ 23 The sparse feature F after fusion with F4 234 ; F 234 After enhancement, we get F 234 Enhanced sparse features F″ 234 ; F′ 234 After channel compression, the output single-scale sparse feature F″ of 3DFC is obtained 234 ;

[0111] The single-scale sparse feature entropy encoding and decoding module is configured as: the output single-scale sparse feature F″ of 3DFC 234 Sort by the same rules as G-PCC to obtain the sorted single-scale sparse features F″ s ; F″ s The geometric coordinates G are geometrically encoded by G-PCC to obtain the geometric coordinate code stream B1, F″ s The characteristic attribute A is entropy encoded by DeepCABAC to obtain the characteristic attribute code stream B2; the geometric coordinate code stream B1 is geometrically encoded by G-PCC to obtain the decoded geometric coordinates The characteristic attribute bitstream B2 is entropy encoded by DeepCABAC to obtain the decoded characteristic attribute and After recombining, the single-scale sparse feature F at the decoding end is obtained dec ;

[0112] The multi-scale sparse feature separation and expansion module is configured as: single-scale sparse feature F at the decoding end dec After three parallel channel upsampling operations, the channel upsampling features are obtained respectively and and After three parallel independent feature point attention modules, we get and After spatial upsampling and enhancement, a sparse feature with a spatial size equal to F3 is obtained. After spatial upsampling and enhancement again, the reconstructed sparse features of F2 are obtained. After spatial upsampling and enhancement, a sparse feature with a spatial size equal to F3 is obtained. and reconstruct sparse features After 3DSE fusion, the reconstructed sparse features of F3 are obtained After enhancement, we get Enhanced sparse features and reconstruct sparse features After 3DSE fusion, the reconstructed sparse features of F4 are obtained.

[0113] The target detection module is configured to: reconstruct multi-scale sparse features and After the 3D target detection algorithm, the final result of 3D target detection is obtained.

[0114] The beneficial effects of the present invention are:

[0115] 1. Different from the existing methods that can only compress 2D image features and 2D bird's-eye view features extracted from 3D point clouds, the present invention proposes a compression encoding method for multi-scale sparse features, which realizes the compression and reconstruction of 3D multi-scale sparse features.

[0116] 2. Compared with the existing methods, the present invention can optimize 3D machine vision tasks and provide a higher compression ratio while ensuring that the task performance does not decrease significantly.

[0117] 3. Since the method proposed in the present invention compresses the potential features extracted by the neural network, only the machine can obtain the useful information therein, thus protecting the privacy of the user.

[0118] 4. Since part of the neural network of the method proposed in the present invention is deployed on the mobile device, the computing pressure of the cloud computing device can be significantly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0119] Figure 1 It is a flow chart of the compression-then-analysis method.

[0120] Figure 2 It is a schematic diagram of the overall process of the method proposed in the present invention.

[0121] Figure 3 It is a schematic diagram of the overall network architecture of the method proposed in the present invention.

[0122] Figure 4 It is a schematic diagram of the network structure of the 3DFC feature fusion module proposed in the present invention.

[0123] Figure 5 Schematic diagram of the network structure of the spatial size downsampling module proposed in the present invention.

[0124] Figure 6 It is a schematic diagram of the network structure of the IRN module.

[0125] Figure 7 Schematic diagram of the network structure of the channel downsampling module proposed in the present invention.

[0126] Figure 8 It is a schematic diagram of the network structure of the 3DSE fusion module proposed in the present invention.

[0127] Fig. 9 Schematic diagram of the network structure of the spatial size upsampling module proposed in the present invention.

[0128] Fig.10It is a schematic diagram of the network structure of the feature point attention module proposed in the present invention.

[0129] Fig.11 Schematic diagram of the network structure of the channel upsampling module proposed in the present invention.

[0130] Fig.12 It is the rate misalignment curve of the present invention and the CTA method based on G-PCC. DETAILED DESCRIPTION

[0131] The present invention will be further defined below in conjunction with the accompanying drawings and embodiments, but is not limited thereto.

[0132] Example 1

[0133] A multi-scale sparse feature compression method based on three-dimensional sparse convolution, such as Figure 2 As shown, including:

[0134] Multi-scale sparse feature extraction, including: Input the LiDAR point cloud P to be detected L After Voxel R-CNN front-end feature extraction, multi-scale sparse features F2, F3 and F4 are obtained;

[0135] The fusion and compression of multi-scale sparse features include: the sparse feature F2 with the largest size in the multi-scale sparse features is compressed in space to obtain a sparse feature F′2 with a space size equal to F3; F″2 is enhanced to obtain a sparse feature F″2 enhanced by F″2; F″2 and F3 are fused by 3DFC features to obtain a sparse feature F after fusion of F″2 and F3 23 ; F 23 After the spatial dimension is compressed, the sparse feature F′ with a spatial dimension equal to F4 is obtained. 23 ; F′ 23 After enhancement, we get F′ 23 Enhanced sparse features F″ 23 ; F″ 23 After 3DFC feature fusion with F4, we get F″ 23 The sparse feature F after fusion with F4 234 ; F 234 After enhancement, we get F 234 Enhanced sparse feature F′ 234 ; F′ 234 After channel compression, the output single-scale sparse feature F″ of 3DFC is obtained 234 ;

[0136] Single-scale sparse feature entropy encoding and decoding, including: 3DFC output single-scale sparse feature F″ 234 Sort by the same rules as G-PCC to obtain the sorted single-scale sparse features F″s ; F″ s The geometric coordinates G are geometrically encoded by G-PCC to obtain the geometric coordinate code stream B1, F″ s The feature attribute A is entropy encoded by DeepCABAC (see S.Wiedemann et al., "DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks," in IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 4, pp. 700-714, May 2020) to obtain the feature attribute code stream B2; the geometric coordinate code stream B1 is geometrically decoded by G-PCC to obtain the decoded geometric coordinates The characteristic attribute bitstream B2 is decoded by DeepCABAC entropy to obtain the decoded characteristic attributes and After recombining, the single-scale sparse feature F at the decoding end is obtained dec ;

[0137] Multi-scale sparse feature separation and expansion, including: single-scale sparse feature F at the decoding end dec After three parallel channel upsampling operations, the channel upsampling features are obtained respectively and and After three feature attention modules, we get and After spatial upsampling and enhancement, a sparse feature with a spatial size equal to F3 is obtained. After spatial upsampling and enhancement again, the reconstructed sparse features of F2 are obtained. After spatial upsampling and enhancement, a sparse feature with a spatial size equal to F3 is obtained. and reconstruct sparse features After 3DSE fusion, the reconstructed sparse features of F3 are obtained After enhancement, we get Enhanced sparse features and reconstruct sparse features After 3DSE fusion, the reconstructed sparse features of F4 are obtained.

[0138] Object detection, including: reconstruction of multi-scale sparse features and After the 3D target detection algorithm, the final result of 3D target detection is obtained.

[0139] Example 2

[0140] A multi-scale sparse feature compression method based on three-dimensional sparse convolution according to embodiment 1 is characterized in that:

[0141] The overall network architecture is as follows Figure 3 As shown; Input LiDAR point cloud P L After the VoXel R-CNN front-end feature extraction network, multi-scale sparse features F2, F3 and F4 are obtained; including:

[0142] The input LiDAR point cloud P of Voxel R-CNN L It is sent to the Voxel R-CNN front-end feature extraction network (reference Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wen gang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnn: Towards high performance voxel-based 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.) to obtain the output multi-scale sparse features F2, F3 and F4; among them, P L Expressed as Where N L represents the number of points in the LiDAR point cloud, 4 represents the four channels of X, Y, Z and Refc, Refc represents the reflectivity, and X, Y, Z represent the three coordinates of the point in the three-dimensional space; each multi-scale sparse feature in F2, F3 and F4 is expressed in the form of formula (I):

[0143]

[0144] In the formula, F represents 3D sparse features, N represents the number of feature points in the feature, (z i ,y i ,xi,b i ) represents the geometric coordinates corresponding to the i-th feature point and the batch to which the i-th feature point belongs. M represents the dimension of the feature attribute of each feature point in the feature. The feature attribute corresponding to the i-th feature point represents the feature, (z max ,y max, x max ) represents the maximum value of the coordinates of each point in the sparse feature, which is the spatial size of the sparse feature. The spatial sizes of F2, F3 and F4 are: [800, 704, 21], [400, 352, 11], [200, 176, 5] respectively.

[0145] The sparse feature F2 with the largest size in the multi-scale sparse features is compressed in space size to obtain a sparse feature F′2 with a space size equal to F3; including:

[0146] Input F2 into the spatial downsampling module to obtain the sparse feature F′2 with a spatial size equal to F3, as shown in formula (II):

[0147] F′2=Add(LReLU(Conv1(LReLU(Subm(F2)))), LReLU(Conv2(F2))) (II)

[0148] Wherein, Subm(·) represents a 3D submanifold sparse convolution with a kernel size of 3 and an output channel number twice the number of input channels (see Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 3dsemantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pages 9224-9232, 2018.); Conv1(·) and Conv2(·) both represent ordinary 3D sparse convolutions with a kernel size of 3, a stride of 2, and an output channel number equal to the input channel number; LReLU(·) represents a LeakyReLU activation function; Add(·) represents a 3D sparse tensor addition operation, as shown in Formula (III) and Formula (IV):

[0149] G o =G m ∪G n (III)

[0150]

[0151] In the formula, G o [i] represents the output sparse feature F o The geometric coordinates corresponding to the i-th feature point; A o [i] represents the output sparse feature Fo The characteristic attribute corresponding to the i-th feature point; G m [i] represents the first input sparse feature F m The geometric coordinates corresponding to the i-th feature point; A m [i] represents the first input sparse feature F m The characteristic attribute corresponding to the i-th feature point; G n [i] represents the second input sparse feature F n The geometric coordinates corresponding to the i-th feature point; A n [i] represents the first input sparse feature F n The feature attribute corresponding to the i-th feature point.

[0152] After F′2 is enhanced by the enhancement module, the sparse feature F″2 enhanced by F′2 is obtained, including:

[0153] F′2 passes through two series-connected IRNBlocks (see J.Wang, D.Ding, Z.Li and Z.Ma, "Multiscale Point Cloud Geometry Compression," 2021 Data Compression Conference (DCC), Snowbird, UT, USA, 2021, pp.73-82.) to aggregate and enhance the features and obtain the enhanced sparse feature F″2. Among them, the output coefficient features of IRNBlock do not change except for the feature attributes, and the number of channels of input and output are the same.

[0154] F″2 and F3 are fused through the 3DFC feature fusion module, such as Figure 4 As shown, the sparse feature F is obtained by fusion of F″2 and F3. 23 ,include:

[0155] The 3DFC feature fusion module is shown in formula (V):

[0156] F 23 =LReLU(Subm3(diffcat(LReLU(Subm2(LReLU(Subm1(F″2))))),F3)) (V)

[0157] In the formula, F 23represents the sparse features after the fusion of F″2 and F3; Subm1(·), Subm2(·), and Subm3(·) all represent 3D submanifold sparse convolutions with a convolution kernel size of 3 and the number of output channels equal to the number of F3 channels; LReLU(·) represents the LeakyReLU activation function; diffcat(·) represents the connection method for 3D sparse features with different geometric coordinate information, as shown in formulas (VI) and (VII):

[0158] G c =G m ∪G n (VI)

[0159]

[0160] In the formula, G c [i] represents the sparse feature F after output addition c The geometric coordinates corresponding to the i-th feature point; A c [i] represents the sparse feature F after output addition c The characteristic attribute corresponding to the i-th feature point; G m [i] represents the first input sparse feature F m The geometric coordinates corresponding to the i-th feature point; A m [i] represents the first input sparse feature F m The characteristic attribute corresponding to the i-th feature point; G n [i] represents the second input sparse feature F n The geometric coordinates corresponding to the i-th feature point; A n [i] represents the first input sparse feature F n The characteristic attribute corresponding to the i-th feature point; cat(·) represents a normal tensor connection operation.

[0161] The feature F obtained by merging F″2 and F3 obtained in the previous step 23 , again through a spatial size downsampling module proposed by the present invention, such as Figure 5 As shown, the spatial size is further reduced to obtain a sparse feature F′ with a spatial size equal to F4 23 .

[0162] F′ 23 Input it again into two IRNBlocks connected in series for enhancement, and we get F′ 23 Enhanced sparse features F″ 23 .

[0163] F″ 23 It is input into the 3DFC feature fusion module proposed by the present invention at the same time as F4 for fusion, and outputs F″ 23The sparse feature F after fusion with F4 234 .

[0164] Then F 234 After being enhanced again by an enhancement module consisting of two IRNBlocks in series, F 234 Enhanced sparse feature F′ 234 .

[0165] F′ 234 After the channel downsampling module, such as Figure 7 As shown, the output single-scale sparse feature F″ of 3DFC is obtained 234 ,include:

[0166] A channel downsampling module with a compression ratio of 32, including two channel compression modules with compression ratios of 4 and 8 respectively; F′ 234 Through the channel downsampling module proposed in the present invention, 32 times of channel downsampling is achieved, and the number of channels is reduced from the original 64 to 2; wherein, the specific architecture of the channel downsampling module is expressed as formula (VIII):

[0167] F″ 234 =Comp(Comp(F′ 234 ,8),4) (VIII)

[0168] In the formula, F″ 234 is the output single-scale sparse feature of 3DFC; Comp(F in , N) represents the N-fold channel compression module, the number of channels of its output features is 1 / N of the number of input feature channels; Comp(F in , N) is represented by formula (IX), (X), (XI):

[0169] F inter1 =LReLU(Subm3(LReLU(Subm2(LReLU(Subm1(F in ))))) (IX)

[0170] F inter2 =LReLU(Subm4(F in ) (X)

[0171] F out =Add(F inter1 , F inter2 ) (XI)

[0172] In formula (IX), F in represents the input features of the channel compression module; F inter1represents the first intermediate feature of the channel compression module; LReLU(·) represents the LeakyReLU activation function; Subm1(·) represents the submanifold sparse convolution with the number of output channels equal to the number of input channels 1 / N, and the convolution kernel size is 1; Subm2(·) represents the submanifold sparse convolution with the number of output channels equal to the number of input channels, and the convolution kernel size is 3; Subm3(·) represents the submanifold sparse convolution with the number of output channels equal to the number of input channels, and the convolution kernel size is 1; In formula (X), F inter2 represents the second intermediate feature of the channel compression module; Subm4(·) represents the submanifold sparse convolution whose output channel number is 1 / N of the input channel number and the convolution kernel size is 3; in formula (XI), F out represents the output features of the channel compression module, and Add(·) represents the 3D sparse tensor addition operation.

[0173] The output single-scale sparse feature F″ of 3DFC 234 Sort by the same rules as G-PCC to obtain the sorted single-scale sparse features F″ s ;include:

[0174] The output single-scale sparse feature F″ of 3DFC 234 The geometric coordinates G″ 234 Arrange the coordinates in the order of x from small to large, and then arrange the coordinates with the same x in the order of y from small to large. After arrangement, the coordinates with the same x and y are arranged in the order of z from small to large, and finally obtain the sorted geometric coordinates G; then according to the one-to-one correspondence between the original geometric coordinates and the feature attributes, 234 The feature attributes are rearranged to obtain the sorted feature attribute A; finally, the geometric coordinates G and the feature attribute A are recombined into the sorted single-scale sparse feature F″ s .

[0175] F″ s The geometric coordinates G are passed through the G-PCC geometric encoder to obtain the geometric coordinate code stream B1, F″ s The characteristic attribute A is passed through the DeepCABAC entropy encoder to obtain the characteristic attribute code stream B2; including:

[0176] F″ s The geometric coordinates G are input into the geometric encoder of G-PCC, and the geometric encoder of G-PCC outputs the geometric coordinate code stream B1; F″ s The feature attribute A is input into the DeepCABAC encoder, and the DeepCABAC encoder outputs the feature attribute code stream B2.

[0177] The geometric coordinate code stream B1 passes through the G-PCC geometry decoder to obtain the decoded geometric coordinates The characteristic attribute bitstream B2 passes through the DeepCABAC entropy decoder to obtain the decoded characteristic attributes and After recombining, the single-scale sparse feature F at the decoding end is obtained dec ,include:

[0178] The geometric coordinate code stream B1 is input into the G-PCC decoder, and the G-PCC decoder outputs the decoded geometric coordinates Then the feature attribute code stream B2 is input into the DeepCABAC entropy decoder, and the DeepCABAC entropy decoder outputs the decoded feature attribute Then and Recombined to obtain the single-scale sparse feature F at the decoding end dec .

[0179] The single-scale sparse feature F at the decoding end dec After three parallel channel upsampling operations, the channel upsampling features are obtained respectively and include:

[0180] Channel upsampling module with upsampling ratio of 32, such as Fig.11 As shown, it includes two channel expansion modules with upsampling ratios of 4 and 8 respectively; Fd e First, two serial channel upsampling modules proposed by the present invention are used to achieve 8-fold and 4-fold upsampling of the channel, respectively. The number of channels changes from 2 to 64, as shown in formula (XII):

[0181]

[0182] In the formula, CExp(Fi n , N) represents an N-fold channel expansion module, and the number of channels of its output features is N times the number of channels of the input features; is the feature after upsampling of the first branch channel; among them, CExp(F in , N) is specifically represented by formula (XIII), (XIV), (XV):

[0183] F inter1 =LReLU(Subm3(LReLU(Subm2(LReLU(Subm1(F in ))))) (XIII)

[0184] F inter2 =LReLU(Subm4(F in ) (XIV)

[0185] Fout =Add(F inter1 , F inter2 ) (XV)

[0186] In formula (XIII), F in represents the input features of the channel expansion module; F inter1 represents the first intermediate feature of the channel expansion module; LReLU(·) represents the LeakyReLU activation function; Subm1(·) represents the submanifold sparse convolution with the number of output channels being N times the number of input channels, and the convolution kernel size is 1; Subm2(·) represents the submanifold sparse convolution with the number of output channels being equal to the number of input channels, and the convolution kernel size is 3; Subm3(·) represents the submanifold sparse convolution with the number of output channels being equal to the number of input channels, and the convolution kernel size is 1; In formula (XIV), F inter2 represents the second intermediate feature of the channel expansion module; Subm4(·) represents the submanifold sparse convolution with the number of output channels being N times the number of input channels and the convolution kernel size being 3; in formula (XV), F out represents the output feature of the channel expansion module, and Add(·) represents the 3D sparse tensor addition operation;

[0187] F dec Perform two channel upsampling at different magnifications to obtain the features after channel upsampling and As shown in formula (XVI) and formula (XVII):

[0188]

[0189] and After three feature attention modules respectively, we get and include:

[0190] Will and The inputs are respectively fed into three parallel independent feature point attention modules, and the features after attention perception are respectively output. and As shown in formula (XVIII), formula (XIX), and formula (XX):

[0191]

[0192] In the formula, Att(·) represents the feature point attention module, such as Fig.10 As shown, the network structure of the feature point attention module is shown in formula (XXI), formula (XXII), formula (XXIII) and formula (XXIV):

[0193] F inter1=Resblock(Resblock(F in )) (XXI)

[0194] F inter2 =Softmax(Subm1(Resblock(Resblock(F in )))) (XXII)

[0195] F out =Add(F in , F inter1 ×F inter2 ) (XXIII)

[0196] F res =Add(F inres ,LReLU(Subm4(LReLU(Subm3(LReLU(Subm2(F inres ))))))) (XXIV)

[0197] In formula (XXI), F in represents the input features of the feature point attention module, F inter1 represents the first intermediate feature of the feature point attention module, Resblock(·) represents the 3D residual convolution block; the specific operation of Resblock(·) is shown in formula (XXIV), where F res represents the output features of the residual block, F inres represents the input features of the residual block, Subm2(·), Subm3(·), and Subm4(·) respectively represent the submanifold sparse convolution with the number of output channels being 1 / 2, 1 / 2, and 1 times the number of input channels, and the convolution kernel size is 3; in formula (XXII), F inter2 is the second intermediate feature of the feature point attention module, Subm1(·) represents the submanifold sparse convolution with an output channel of 1 and a convolution kernel size of 5, and Softmax(·) represents the Softmax function. In formula (XXIII), Add(·) represents the 3D sparse tensor addition operation.

[0198] After the spatial size upsampling module and the enhancement module, a sparse feature with a spatial size equal to F3 is obtained. include:

[0199] Input is sent to a module consisting of two IRNs and a spatial size upsampling module, such as Fig. 9 As shown, the output space size is equal to the sparse feature of F3 As shown in formula (XXV) and formula (XXVI):

[0200]

[0201] F out =Add(LReLU(Conv1(LReLU(Subm(F in )))), LReLU(Conv2(F in )))) (XXVI)

[0202] Where IRN(·) represents the IRN module; Subm(·) represents a 3D submanifold sparse volume with a convolution kernel size of 3 and an output channel number twice the number of input channels; UPScale(·) represents the spatial size upsampling module, as shown in formula (XXVI), Fi n Represents the sparse features of the input; F out Represents the output sparse features; Conv1(·) and Conv2(·) both represent transposed sparse convolution, the convolution kernel size is 3, the stride is 1, the number of output channels is equal to the number of input channels, and the size of the output sparse features is twice the size of the input sparse features; LReLU(·) represents the LeakyReLU activation function; Add(·) represents the 3D sparse tensor addition operation.

[0203] After the spatial size upsampling module and the enhancement module again, the reconstructed sparse features of F2 are obtained. include:

[0204] It is input again into two IRNs and a spatial size upsampling module, and the output spatial size is equal to the sparse features of F2 As shown in formula (XXVII):

[0205]

[0206] After spatial upsampling and enhancement modules, a sparse feature with a spatial size equal to F3 is obtained. include:

[0207] Input is a sparse feature of two IRNs and a spatial size upsampling module, and the output spatial size is equal to F3 As shown in formula (XXVIII):

[0208]

[0209] and reconstruct sparse features After 3DSE fusion module, such as Figure 8 As shown, the reconstructed sparse features of F3 are obtained include:

[0210] and Input into the 3DSE fusion module and output the reconstructed sparse features of F3 As shown in formula (XXIX):

[0211]

[0212] In the formula, represent and The fused sparse features, i.e. the reconstructed sparse features of F3 Subm1(·) and Subm2(·) both represent a convolution kernel size of 3, and the number of output channels is equal to 3D submanifold sparse convolution with the same number of channels; Conv1(·) represents 3D sparse convolution with a kernel size of 3, a stride of 2, and the number of output channels equal to the number of input channels; LReLU(·) represents the LeakyReLU activation function; diffcat(·) represents the connection method for 3D sparse features with different geometric coordinate information;

[0213] After the enhancement module, we get Enhanced sparse features include:

[0214] Input to two IRN modules connected in series, output Enhanced sparse features As shown in formula (XXX):

[0215]

[0216] and reconstruct sparse features After the 3DSE fusion module, the reconstructed sparse features of F4 are obtained. include:

[0217] and Input into the 3DSE fusion module, output the reconstructed sparse features of F4 As shown in formula (XXXI):

[0218]

[0219] In the formula, represent and The fused sparse features, i.e. the reconstructed sparse features of F4 Subm1(·) and Subm2(·) both represent a convolution kernel size of 3 and an output channel number equal to The number of channels of the 3D submanifold sparse convolution is 3D; Conv1(·) represents a 3D sparse convolution with a kernel size of 3, a stride of 2, and the number of output channels equal to the number of input channels; LReLU(·) represents the LeakyReLU activation function; diffcat(·) represents the connection method for 3D sparse features with different geometric coordinate information.

[0220] Reconstructing multi-scale sparse features and After the back-end network of the 3D object detection algorithm Voxel R-CNN (reference Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wen gang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnn: Towards high performance voxel-based 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.), the final result of 3D object detection is obtained; as shown in formula (XXXII):

[0221]

[0222] In the formula, R Det represents the final result of 3D object detection; TaskNetwork(·) represents the backend task network of Voxel R-CNN.

[0223] Fig.12 It is the rate misalignment curve of the present invention and the CTA method based on G-PCC.

[0224] Example 3

[0225] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the multi-scale sparse feature compression method based on three-dimensional sparse convolution described in Example 1 or 2 are implemented.

[0226] Example 4

[0227] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the multi-scale sparse feature compression method based on three-dimensional sparse convolution described in Example 1 or 2.

[0228] Example 5

[0229] A multi-scale sparse feature compression system based on three-dimensional sparse convolution, comprising:

[0230] The multi-scale sparse feature extraction module is configured as follows: Input the LiDAR point cloud P to be detected L After Voxel R-CNN front-end feature extraction, multi-scale sparse features F2, F3 and F4 are obtained;

[0231] The multi-scale sparse feature fusion and compression module is configured as follows: the sparse feature F2 with the largest size in the multi-scale sparse features is compressed in spatial size to obtain a sparse feature F′2 with a spatial size equal to F3; F′2 is enhanced to obtain a sparse feature F″2 enhanced by F′2; F″2 and F3 are fused by 3DFC features to obtain a sparse feature F′2 fused by F3. 23 ; F 23 After the spatial dimension is compressed, the sparse feature F′ with a spatial dimension equal to F4 is obtained. 23 ; F″ 23 After enhancement, we get F′ 23 Enhanced sparse features F″ 23 ; F″ 23 After 3DFC feature fusion with F4, we get F″ 23 The sparse feature F after fusion with F4 234 ; F 234 After enhancement, we get F 234 Enhanced sparse feature F′ 234 ; F″ 234 After channel compression, the output single-scale sparse feature F″ of 3DFC is obtained 234 ;

[0232] The single-scale sparse feature entropy encoding and decoding module is configured as: the output single-scale sparse feature F″ of 3DFC 234 Sort by the same rules as G-PCC to obtain the sorted single-scale sparse features F″ s ; F″ s The geometric coordinates G are geometrically encoded by G-PCC to obtain the geometric coordinate code stream B1, F″ sThe feature attribute A is entropy encoded by DeepCABAC (see S.Wiedemann et al., "DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks," in IEEE Journal of Selected Topics in Signal Processing, vol.14, no.4, pp.700-714, May 2020.) to obtain the feature attribute code stream B2; the geometric coordinate code stream B1 is geometrically encoded by G-PCC to obtain the decoded geometric coordinates The characteristic attribute bitstream B2 is entropy encoded by DeepCABAC to obtain the decoded characteristic attribute and After recombining, the single-scale sparse feature F at the decoding end is obtained dec ;

[0233] The multi-scale sparse feature separation and expansion module is configured as: single-scale sparse feature F at the decoding end dec After three parallel channel upsampling operations, the channel upsampling features are obtained respectively and and After three parallel independent feature point attention modules, we get and After spatial upsampling and enhancement, a sparse feature with a spatial size equal to F3 is obtained. After spatial upsampling and enhancement again, the reconstructed sparse features of F2 are obtained. After spatial upsampling and enhancement, a sparse feature with a spatial size equal to F3 is obtained. and reconstruct sparse features After 3DSE fusion, the reconstructed sparse features of F3 are obtained After enhancement, we get Enhanced sparse features and reconstruct sparse features After 3DSE fusion, the reconstructed sparse features of F4 are obtained.

[0234] The target detection module is configured to: reconstruct multi-scale sparse features and After the 3D target detection algorithm, the final result of 3D target detection is obtained.

Claims

1. A multi-scale sparse feature compression method based on three-dimensional sparse convolution, characterized in that: include: Multi-scale sparse feature extraction, including: Input the LiDAR point cloud P to be detected L After Voxel R-CNN front-end feature extraction, multi-scale sparse features F2, F3 and F4 are obtained; The fusion and compression of multi-scale sparse features include: the sparse feature F2 with the largest size in the multi-scale sparse features is compressed in space to obtain a sparse feature F″2 with a space size equal to F3; F″2 is enhanced to obtain a sparse feature F″2 enhanced by F′2; F″2 and F3 are fused by 3DFC features to obtain a sparse feature F after fusion of F″2 and F3 23 ; F 23 After the spatial dimension is compressed, the sparse feature F″ with a spatial dimension equal to F4 is obtained. 23 ; F′ 23 After enhancement, we get F′ 23 Enhanced sparse features F″ 23 ; F″ 23 After 3DFC feature fusion with F4, we get F″ 23 The sparse feature F after fusion with F4 234 ; F 234 After enhancement, we get F 234 Enhanced sparse feature F′ 234 ; F′ 234 After channel compression, the output single-scale sparse feature F′ of 3DFC is obtained 234 ; Single-scale sparse feature entropy encoding and decoding, including: 3DFC output single-scale sparse feature F″ 234 Sort by the same rules as G-PCC to obtain the sorted single-scale sparse features F″ s ; F″ s The geometric coordinates G are geometrically encoded by G-PCC to obtain the geometric coordinate code stream B1, F″ s The characteristic attribute A is entropy encoded by DeepCABAC to obtain the characteristic attribute bitstream B2; the geometric coordinate bitstream B1 is geometrically decoded by G-PCC to obtain the decoded geometric coordinates The characteristic attribute bitstream B2 is decoded by DeepCABAC entropy to obtain the decoded characteristic attributes and After recombining, the single-scale sparse feature F at the decoding end is obtained dec ; Multi-scale sparse feature separation and expansion, including: single-scale sparse feature F at the decoding end dec After three parallel channel upsampling operations, the channel upsampling features are obtained respectively and and After three feature attention modules, we get and After spatial upsampling and enhancement, a sparse feature with a spatial size equal to F3 is obtained. After spatial upsampling and enhancement again, the reconstructed sparse features of F2 are obtained. After spatial upsampling and enhancement, a sparse feature with a spatial size equal to F3 is obtained. and reconstruct sparse features After 3DSE fusion, the reconstructed sparse features of F3 are obtained After enhancement, we get Enhanced sparse features and reconstruct sparse features After 3DSE fusion, the reconstructed sparse features of F4 are obtained. Object detection, including: reconstruction of multi-scale sparse features and After the 3D target detection algorithm, the final result of 3D target detection is obtained.

2. The multi-scale sparse feature compression method based on three-dimensional sparse convolution according to claim 1, characterized in that: Input LiDAR point cloud P L After the Voxel R-CNN front-end feature extraction network, multi-scale sparse features F2, F3 and F4 are obtained; include: The input LiDAR point cloud P of VoxelR-CNN L It is sent to the VoxelR-CNN front-end feature extraction network to obtain the output multi-scale sparse features F2, F3 and F4; among them, P L Expressed as Where N L represents the number of points in the LiDAR point cloud, 4 represents the four channels of X, Y, Z and Refc, Refc represents the reflectivity, and X, Y, Z represent the three coordinates of the point in the three-dimensional space; each multi-scale sparse feature in F2, F3 and F4 is expressed in the form of formula (I): In the formula, F represents 3D sparse features, N represents the number of feature points in the feature, (z i ,y i , x i , b i ) represents the geometric coordinates corresponding to the i-th feature point and the batch to which the i-th feature point belongs. M represents the dimension of the feature attribute of each feature point in the feature. The feature attribute corresponding to the i-th feature point represents the feature, (z max ,y max , x max ) represents the maximum possible value of the coordinates of each point in the sparse feature, which is the spatial size of the sparse feature.

3. The multi-scale sparse feature compression method based on three-dimensional sparse convolution according to claim 1, characterized in that: The sparse feature F2 with the largest size among the multi-scale sparse features is compressed in spatial size to obtain the sparse feature F′2 with spatial size equal to F3; include: Input F2 into the spatial downsampling module to obtain the sparse feature F″2 with a spatial size equal to F3, as shown in formula (II): F′2=Add(LReLU(Conv1(LReLU(Subm(F2)))), LReLU(Conv2(F2))) (II) Where Subm(·) represents a 3D submanifold sparse convolution with a kernel size of 3 and an output channel number twice the number of input channels. Conv1(·) and Conv2(·) both represent ordinary 3D sparse convolutions with a kernel size of 3, a stride of 2, and an output channel number equal to the input channel number. LReLU(·) represents the LeakyReLU activation function. Add(·) represents a 3D sparse tensor addition operation, as shown in Equation (III) and Equation (IV): G o =G m ∪G n (III) In the formula, G o [i] represents the output sparse feature F o The geometric coordinates corresponding to the i-th feature point; A o [i] represents the output sparse feature F o The characteristic attribute corresponding to the i-th feature point; G m [i] represents the first input sparse feature F m The geometric coordinates corresponding to the i-th feature point; A m [i] represents the feature attribute corresponding to the i-th feature point of the first input sparse feature Fm; G n [i] represents the second input sparse feature F n The geometric coordinates corresponding to the i-th feature point; A n [i] represents the first input sparse feature F n The feature attribute corresponding to the i-th feature point; Further preferably, F′2 is subjected to an enhancement module to obtain a sparse feature F″2 enhanced by F″2, including: F′2 is aggregated and enhanced through two serial IRNBlocks to obtain the enhanced sparse feature F″2; Further preferably, F″2 and F3 are fused by a 3DFC feature fusion module to obtain a sparse feature F after fusion of F″2 and F3. 23 ,include: The 3DFC feature fusion module is shown in formula (V): F 23 =LReLU(Subm3(diffcat(LReLU(Subm2(LReLU(Subm1(F″2))))),F3)) (V) In the formula, F 23 represents the sparse features after the fusion of F″2 and F3; Subm1(·), Subm2(·), Subm3(·) all represent 3D submanifold sparse convolution with a convolution kernel size of 3 and the number of output channels equal to the number of F3 channels; LReLU(·) represents the LeakyReLU activation function; diffcat(·) represents the connection method for 3D sparse features with different geometric coordinate information, as shown in formula (VI) and (VII): G c =G m ∪G n (VI) In the formula, G c [i] represents the sparse feature F after output addition c The geometric coordinates corresponding to the i-th feature point; A c [i] represents the sparse feature F after output addition c The characteristic attribute corresponding to the i-th feature point; G m [i] represents the first input sparse feature F m The geometric coordinates corresponding to the i-th feature point; A m [i] represents the first input sparse feature F m The characteristic attribute corresponding to the i-th feature point; G n [i] represents the second input sparse feature F n The geometric coordinates corresponding to the i-th feature point; A n [i] represents the first input sparse feature F n The characteristic attribute corresponding to the i-th feature point; cat(·) represents a common tensor connection operation; More preferably, F′ 234 After the channel downsampling module, the output single-scale sparse feature F″ of 3DFC is obtained 234 ,include: F′ 234 After the channel downsampling module, the channel is downsampled by 32 times, and the number of channels is reduced from the original 64 to 2. The specific architecture of the channel downsampling module is expressed as formula (VIII): F″ 234 =Comp(Comp(F′ 234 ,8),4) (VIII) In the formula, F″ 234 is the output single-scale sparse feature of 3DFC; Comp(F in , N) represents the N-fold channel compression module, the number of channels of its output features is 1 / N of the number of input feature channels; Comp(F in , N) is represented by formula (IX), (X), (XI): F inter1 =LReLU(Subm3(LReLU(Subm2(LReLU(Subm1(F in ))))) (IX) F inter2 =LReLU(Subm4(Fi n ) (X) F out =Add(F inter1 ,F inter2 ) (XI) In formula (IX), F in represents the input features of the channel compression module; F inter1 represents the first intermediate feature of the channel compression module; LReLU(·) represents the LeakyReLU activation function; Subm1(·) represents the submanifold sparse convolution with the number of output channels equal to the number of input channels 1 / N, and the convolution kernel size is 1; Subm2(·) represents the submanifold sparse convolution with the number of output channels equal to the number of input channels, and the convolution kernel size is 3; Subm3(·) represents the submanifold sparse convolution with the number of output channels equal to the number of input channels, and the convolution kernel size is 1; In formula (X), F inter2 represents the second intermediate feature of the channel compression module; Subm4(·) represents the submanifold sparse convolution whose output channel number is 1 / N of the input channel number and the convolution kernel size is 3; in formula (XI), F out represents the output features of the channel compression module, and Add(·) represents the 3D sparse tensor addition operation.

4. The multi-scale sparse feature compression method based on three-dimensional sparse convolution according to claim 1, characterized in that: The output single-scale sparse feature F″ of 3DFC 234 Sort by the same rules as G-PCC to obtain the sorted single-scale sparse features F″ s ;include: The output single-scale sparse feature F″ of 3DFC 234 The geometric coordinates G″ 234 Arrange the coordinates in the order of x from small to large, and then arrange the coordinates with the same x in the order of y from small to large. After arrangement, the coordinates with the same x and y are arranged in the order of z from small to large, and finally obtain the sorted geometric coordinates G; then according to the one-to-one correspondence between the original geometric coordinates and the feature attributes, 234 The feature attributes are rearranged to obtain the sorted feature attribute A; finally, the geometric coordinates G and the feature attribute A are recombined into the sorted single-scale sparse feature F″ s ; Further preferably, F″ s The geometric coordinates G are passed through the G-PCC geometric encoder to obtain the geometric coordinate code stream B1, F″ s The characteristic attribute A is passed through the DeepCABAC entropy encoder to obtain the characteristic attribute code stream B2; including: The single-scale sparse feature F at the decoding end dec After three parallel channel upsampling operations, the channel upsampling features are obtained respectively and include: F dec First, two serial channel upsampling modules are used to upsample the channel by 8 times and 4 times respectively. The number of channels changes from 2 to 64, as shown in formula (XII): In the formula, CExp(F in , N) represents an N-fold channel expansion module, and the number of channels of its output features is N times the number of channels of the input features; is the feature after upsampling of the first branch channel; among them, CExp(F in , N) is specifically represented by formula (XIII), (XIV), (XV): F inter1 =LReLU(Subm3(LReLU(Subm2(LReLU(Subm1(F in ))))) (XIII) F inter2 =LReLU(Subm4(F in ) (XIV) F out =Add(F inter1 ,F inter2 ) (XV) In formula (XIII), F in represents the input features of the channel expansion module; F inter1 represents the first intermediate feature of the channel expansion module; LReLU(·) represents the LeakyReLU activation function; Subm1(·) represents the submanifold sparse convolution with the number of output channels being N times the number of input channels, and the convolution kernel size is 1; Subm2(·) represents the submanifold sparse convolution with the number of output channels being equal to the number of input channels, and the convolution kernel size is 3; Subm3(·) represents the submanifold sparse convolution with the number of output channels being equal to the number of input channels, and the convolution kernel size is 1; In formula (XIV), F inter2 represents the second intermediate feature of the channel expansion module; Subm4(·) represents the submanifold sparse convolution with the number of output channels being N times the number of input channels and the convolution kernel size being 3; in formula (XV), F out represents the output feature of the channel expansion module, and Add(·) represents the 3D sparse tensor addition operation; F dec Perform two channel upsampling at different magnifications to obtain the features after channel upsampling and As shown in formula (XVI) and formula (XVII):

5. The multi-scale sparse feature compression method based on three-dimensional sparse convolution according to claim 1, characterized in that: and After three feature attention modules respectively, we get and include: Will and The inputs are respectively fed into three parallel independent feature point attention modules, and the features perceived by the attention are output respectively. and As shown in formula (XVIII), formula (XIX), and formula (XX): Where Att(·) represents the feature point attention module. The network structure of the feature point attention module is shown in formula (XXI), formula (XXII), formula (XXIII) and formula (XXIV): F inter1 =Resblock(Resblock(F in )) (XXI) F inter2 =Softmax(Subm1(Resblock(Resblock(F in )))) (XXII) F out =Add(F in ,F inter1 ×F inter2 ) (XXIII) F res =Add(F inres ,LReLU(Subm4(LReLU(Subm3(LReLU(Subm2(F inres ))))))) (XXIV) In formula (XXI), F in represents the input features of the feature point attention module, F inter1 represents the first intermediate feature of the feature point attention module, Resblock(·) represents the 3D residual convolution block; the specific operation of Resblock(·) is shown in formula (XXIV), where F res represents the output features of the residual block, F inres represents the input features of the residual block, Subm2(·), Subm3(·), and Subm4(·) respectively represent the submanifold sparse convolution with the number of output channels being 1 / 2, 1 / 2, and 1 times the number of input channels, and the convolution kernel size is 3; in formula (XXII), F inter2 is the second intermediate feature of the feature point attention module, Subm1(·) represents the submanifold sparse convolution with an output channel of 1 and a convolution kernel size of 5, and Softmax(·) represents the Softmax function; in formula (XXIII), Add(·) represents the 3D sparse tensor addition operation; More preferably, After the spatial size upsampling module and the enhancement module, a sparse feature with a spatial size equal to F3 is obtained. include: Input is a sparse feature of two IRNs and a spatial size upsampling module, and the output spatial size is equal to F3 As shown in formula (XXV) and formula (XXVI): F out =Add(LReLU(Conv1(LReLU(Subm(F in )))),LReLU(Conv2(F in )))) (XXVI) Where IRN(·) represents the IRN module; Subm(·) represents a 3D submanifold sparse volume with a convolution kernel size of 3 and an output channel number twice the number of input channels; UPScale(·) represents the spatial size upsampling module, as shown in formula (XXVI), Fi n Represents the sparse features of the input; F out Represents the output sparse features; Conv1(·) and Conv2(·) both represent transposed sparse convolution, the convolution kernel size is 3, the stride is 1, the number of output channels is equal to the number of input channels, and the size of the output sparse features is twice the size of the input sparse features; LReLU(·) represents the LeakyReLU activation function; Add(·) represents the 3D sparse tensor addition operation; After the spatial size upsampling module and the enhancement module again, the reconstructed sparse features of F2 are obtained. include: It is input again into two IRNs and a spatial size upsampling module, and the output spatial size is equal to the sparse features of F2 As shown in formula (XXVII): After spatial upsampling and enhancement modules, a sparse feature with a spatial size equal to F3 is obtained. include: Input is a sparse feature of two IRNs and a spatial size upsampling module, and the output spatial size is equal to F3 As shown in formula (XXVIII):

6. The multi-scale sparse feature compression method based on three-dimensional sparse convolution according to claim 1, characterized in that: and reconstruct sparse features After the 3DSE fusion module, the reconstructed sparse features of F3 are obtained. include: and Input to the 3DSE fusion module and output the reconstructed sparse features of F3 As shown in formula (XXIX): In the formula, represent and The fused sparse features, i.e. the reconstructed sparse features of F3 Subm1(·) and Subm2(·) both represent a convolution kernel size of 3, and the number of output channels is equal to 3D submanifold sparse convolution with the same number of channels; Conv1(·) represents 3D sparse convolution with a kernel size of 3, a stride of 2, and the number of output channels equal to the number of input channels; LReLU(·) represents the LeakyReLU activation function; diffcat(·) represents the connection method for 3D sparse features with different geometric coordinate information; After the enhancement module, we get Enhanced sparse features include: Input to two IRN modules connected in series, as shown in Figure 6, the output Enhanced sparse features As shown in formula (XXX): More preferably, and reconstruct sparse features After the 3DSE fusion module, the reconstructed sparse features of F4 are obtained. include: and Input into the 3DSE fusion module, output the reconstructed sparse features of F4 As shown in formula (XXXI): In the formula, represent and The fused sparse features, i.e. the reconstructed sparse features of F4 Subm1(·) and Subm2(·) both represent a convolution kernel size of 3 and an output channel number equal to The number of channels of the 3D submanifold sparse convolution is 3D; Conv1(·) represents a 3D sparse convolution with a kernel size of 3, a stride of 2, and the number of output channels equal to the number of input channels; LReLU(·) represents the LeakyReLU activation function; diffcat(·) represents the connection method for 3D sparse features with different geometric coordinate information.

7. The multi-scale sparse feature compression method based on three-dimensional sparse convolution according to claim 1, characterized in that: Reconstructing multi-scale sparse features and After the back-end network of the 3D object detection algorithm Voxel R-CNN, the final result of 3D object detection is obtained; as shown in formula (XXXII): In the formula, R Det represents the final result of 3D object detection; TaskNetwork(·) represents the backend task network of VoxelR-CNN.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the multi-scale sparse feature compression method based on three-dimensional sparse convolution described in any one of claims 1-7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-scale sparse feature compression method based on three-dimensional sparse convolution described in any one of claims 1 to 7 are implemented.

10. A multi-scale sparse feature compression system based on three-dimensional sparse convolution, characterized in that: include: The multi-scale sparse feature extraction module is configured as follows: Input the LiDAR point cloud P to be detected L After Voxel R-CNN front-end feature extraction, multi-scale sparse features F2, F3 and F4 are obtained; The multi-scale sparse feature fusion and compression module is configured as follows: the sparse feature F2 with the largest size in the multi-scale sparse features is compressed in space to obtain a sparse feature F′2 with a space size equal to F3; F′2 is enhanced to obtain a sparse feature F″2 enhanced by F′2; F″2 and F3 are fused by 3DFC features to obtain a sparse feature F′2 fused by F3. 23 ; F 23 After the spatial dimension is compressed, the sparse feature F″ with a spatial dimension equal to F4 is obtained. 23 ; F″ 23 After enhancement, we get F′ 23 Enhanced sparse features F″ 23 ; F″ 23 After 3DFC feature fusion with F4, we get F″ 23 The sparse feature F after fusion with F4 234 ; F 234 After enhancement, we get F 234 Enhanced sparse feature F′ 234 ; F″ 234 After channel compression, the output single-scale sparse feature F″ of 3DFC is obtained 234 ; The single-scale sparse feature entropy encoding and decoding module is configured as: the output single-scale sparse feature F″ of 3DFC 234 Sort by the same rules as G-PCC to obtain the sorted single-scale sparse features F″ s ; F″ s The geometric coordinates G are geometrically encoded by G-PCC to obtain the geometric coordinate code stream B1, F″ s The characteristic attribute A is entropy encoded by DeepCABAC to obtain the characteristic attribute code stream B2; the geometric coordinate code stream B1 is geometrically encoded by G-PCC to obtain the decoded geometric coordinates The characteristic attribute bitstream B2 is entropy encoded by DeepCABAC to obtain the decoded characteristic attribute and After recombining, the single-scale sparse feature F at the decoding end is obtained dec ; The multi-scale sparse feature separation and expansion module is configured as: single-scale sparse feature F at the decoding end dec After three parallel channel upsampling operations, the channel upsampling features are obtained respectively and and After three parallel independent feature point attention modules, we get and After spatial upsampling and enhancement, a sparse feature with a spatial size equal to F3 is obtained. After spatial upsampling and enhancement, the reconstructed sparse features of F2 are obtained. After spatial upsampling and enhancement, a sparse feature with a spatial size equal to F3 is obtained. and reconstruct sparse features After 3DSE fusion, the reconstructed sparse features of F3 are obtained After enhancement, we get Enhanced sparse features and reconstruct sparse features After 3DSE fusion, the reconstructed sparse features of F4 are obtained. The target detection module is configured to: reconstruct multi-scale sparse features and After the 3D target detection algorithm, the final result of 3D target detection is obtained.

Images