A lightweight three-dimensional target detection method and device based on sparse convolution pruning

Abstract

The application provides a lightweight three-dimensional target detection method and device based on sparse convolution pruning, and relates to the technical field of automatic driving, comprising: voxelizing original point cloud data to obtain a plurality of voxels; processing the plurality of voxels by using a convolution block to obtain a first feature map; processing the first feature map by using K sequentially connected down-sampling sparse convolution blocks to sequentially obtain K down-sampled feature maps; copying the (K+1)th feature map to obtain the (K+2)th feature map; processing the (K+2)th feature map by using K sequentially connected up-sampling sparse convolution blocks to sequentially obtain K up-sampled feature maps; adding each corresponding element of two feature maps of the same scale in the 2K feature maps to obtain K final feature maps; processing the K final feature maps by using K detection heads of different scales respectively to obtain a three-dimensional target detection result. The application can effectively reduce the calculation amount of sparse convolution and realize lightweight 3D target detection.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a lightweight 3D target detection method and apparatus based on sparse convolution pruning. Background Technology

[0002] Due to the disordered, irregular, and non-uniform nature of radar point clouds, convolutional operations cannot be directly applied to them. Currently, voxel-based methods can be used to process irregular point cloud data. These methods first convert the point cloud into regular voxels and then use mature convolutional neural networks for feature extraction. However, computational costs and memory requirements increase with voxel resolution, making it impractical to train voxel-based models using high-resolution input. Since 3D sparse convolutions are predominantly sparse, background points with less information occupy the main area of ​​the scene, with only a small portion containing voxelized foreground data. Therefore, 3D sparse convolutions can effectively avoid potential redundancy in the model, improving efficiency without sacrificing accuracy. Besides the redundancy in radar point cloud data, the structural design of the 3D sparse convolution model itself can also introduce redundancy. Typically, 3D sparse convolutions use downsampling to extract point cloud features, but downsampling may increase the number of non-empty voxels instead of decreasing them, undoubtedly increasing unnecessary computational costs in subsequent stages. Summary of the Invention

[0003] In view of this, this application provides a lightweight 3D target detection method and apparatus based on sparse convolution pruning to solve the above-mentioned technical problems.

[0004] In a first aspect, embodiments of this application provide a lightweight 3D target detection method based on sparse convolution pruning, including:

[0005] The original point cloud data is voxelized to obtain multiple voxels;

[0006] The first feature map is obtained by processing multiple voxels using pre-trained convolutional blocks.

[0007] The first feature map is processed by K pre-trained, sequentially connected downsampled sparse convolutional blocks to obtain K downsampled feature maps in sequence: the second feature map, the third feature map, ... and the (K+1)th feature map;

[0008] The (K+1)th feature map is copied to obtain the (K+2)th feature map; the (K+2)th feature map is processed by K pre-trained sequentially connected upsampled sparse convolutional blocks to obtain K upsampled feature maps in sequence: the (K+3)th feature map, the (K+4)th feature map, ... and the 2Kth feature map;

[0009] Add each corresponding element of two feature maps of the same scale in the 2K feature maps to obtain K final feature maps; process the K final feature maps with K pre-trained detection heads of different scales to obtain the 3D object detection result.

[0010] Furthermore, the original point cloud data is voxelized to obtain multiple voxels; including:

[0011] The original 3D point cloud data is voxelized using cuboid voxels of a preset size, generating V 3D voxels:

[0012] V=N D ×N H ×N w

[0013] N D =D / V D N H =H / V H N W =W / V W

[0014] Among them, V D V H and V W Here are the three dimensions of the cuboid voxel; D, H, and W are the depth, height, and width of the original 3D point cloud data, and N is the depth, height, and width of the voxel. D N H and N W The number of voxels in the three directions.

[0015] Furthermore, the downsampling sparse convolutional block includes: a feature map partitioning module and a downsampling module; the feature map partitioning module is used to divide each cube with feature values ​​in the input 3D feature map into important cubes P. im and unimportant cube P nim Utilizing all important cubes P im Generate an important feature map of the same size as the input 3D feature map, utilizing all unimportant cubes P. nim The module generates an unimportant feature map of the same size as the input 3D feature map. The downsampling module consists of two parallel first sparse convolutional pruning layers and a downsampling processing unit, as well as a first concatenation unit. The first sparse convolutional pruning layer downsamples the important feature map to obtain a first downsampled feature map. The downsampling processing unit performs pooling on the feature map of the unimportant feature map to obtain a second downsampled feature map, which is the same size as the first downsampled feature map. The first concatenation unit concatenates the first and second downsampled feature maps together to generate the output 3D feature map.

[0016] Furthermore, each cube with eigenvalues ​​in the input 3D feature map is divided into importance cubes P. im and unimportant cube P nim ;include:

[0017] Calculate the mean of the features of all cubes in the input 3D feature map;

[0018] If a cube's feature is greater than the mean of all cubes' features, then that cube is considered an important cube, P. im Otherwise, the cube is an unimportant cube P. nim .

[0019] Furthermore, the input to the k-th downsampled sparse convolutional block is of size (W) k H k D k C k The k-th feature map of ) W k H k and D k Let C be the width, height, and depth of the k-th feature map. k Let K be the dimension of the k-th feature map, 1≤k≤K; the output of the k-th downsampled sparse convolutional block is of size (W). k+1 H k+1 D k+1 C k+1 The (k+1)th feature map of ) W k+1 H k+1 and D k+1 Let C be the width, height, and depth of the (k+1)th feature map. k+1 Let be the dimension of the (k+1)th feature map; the number of convolutional kernels in the first sparse convolutional pruning layer of the k-th downsampled sparse convolutional block is n, the dimension of the convolutional kernels is w*w*d, the stride is s, and the padding value is p; then we have:

[0020]

[0021]

[0022]

[0023] C k+1 =n.

[0024] Furthermore, the upsampling sparse convolutional block includes: a feature map partitioning module and an upsampling module; the feature map partitioning module is used to divide each cube with feature values ​​in the input 3D feature map into important cubes P. im and unimportant cube P nim Utilizing all important cubes P imGenerate an important feature map of the same size as the input 3D feature map, utilizing all unimportant cubes P. nim The module generates an unimportant feature map of the same size as the input 3D feature map. The upsampling module consists of two parallel second sparse convolutional pruning layers, an upsampling processing unit, and a second concatenation unit. The second sparse convolutional pruning layer upsamples the important feature map to obtain a first upsampled feature map. The upsampling processing unit interpolates the unimportant feature map to obtain a second upsampled feature map, which is the same size as the first upsampled feature map. The second concatenation unit concatenates the first and second upsampled feature maps to generate the output 3D feature map.

[0025] Furthermore, the input to the k-th upsampled sparse convolutional block is of size (W) k H k D k C k The k-th feature map of ) W k H k and D k Let C be the width, height, and depth of the k-th feature map. k Let K be the dimension of the k-th feature map, K+2≤k≤2K; the output of the k-th downsampled sparse convolutional block is of size (W). k+1 H k+1 D k+1 C k+1 The (k+1)th feature map of ) W k+1 H k+1 and D k+1 Let C be the width, height, and depth of the (k+1)th feature map. k+1 Let be the dimension of the (k+1)th feature map; the number of convolutional kernels in the second sparse convolutional pruning layer of the kth upsampled sparse convolutional block is n, the dimension of the convolutional kernels is w*w*d, the stride is s, and the padding value is p, then we have:

[0026] W k+1 =(W k -1)*s-2*p+w

[0027] H k+1 =(H k -1)*s-2*p+w

[0028] D k+1 =(D k -1)*s-2*p+d

[0029] C k+1 =n.

[0030] Furthermore, the method also includes the step of jointly training the convolutional block, K downsampled sparse convolutional blocks, K upsampled sparse convolutional blocks, and K detection heads.

[0031] Secondly, embodiments of this application provide a lightweight 3D target detection device based on sparse convolution pruning, comprising:

[0032] The voxel processing unit is used to voxelize the raw point cloud data to obtain multiple voxels.

[0033] The processing unit is used to process multiple voxels using pre-trained convolutional blocks to obtain the first feature map;

[0034] The downsampling unit is used to process the first feature map using K pre-trained, sequentially connected downsampling sparse convolutional blocks to obtain K downsampling feature maps in sequence: the second feature map, the third feature map, ... and the (K+1)th feature map;

[0035] The upsampling unit is used to copy the (K+1)th feature map to obtain the (K+2)th feature map; the (K+2)th feature map is processed by K pre-trained sequentially connected upsampling sparse convolutional blocks to obtain K upsampling feature maps in sequence: the (K+3)th feature map, the (K+4)th feature map, ... and the 2Kth feature map;

[0036] The detection unit is used to add each corresponding element of two feature maps of the same scale in the 2K feature maps to obtain K final feature maps; the K final feature maps are processed by K pre-trained detection heads of different scales to obtain the three-dimensional target detection result.

[0037] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of embodiments of this application.

[0038] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method of embodiments of this application.

[0039] This application can effectively reduce the computational cost of sparse convolution, enabling lightweight 3D object detection. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0041] Figure 1 A flowchart of a lightweight 3D target detection method based on sparse convolution pruning provided in an embodiment of this application;

[0042] Figure 2 A schematic diagram of the eight feature maps provided in the embodiments of this application;

[0043] Figure 3 A functional structure diagram of a lightweight 3D target detection device based on sparse convolution pruning provided in an embodiment of this application;

[0044] Figure 4 A functional structure diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0046] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0047] First, a brief introduction to the design concept of the embodiments of this application will be given.

[0048] 3D object detection is an important research area in computer vision, with wide applications in autonomous driving, robotic environmental perception, and environmental exploration. However, because the majority of data within a 3D sparse convolution is sparse, background points with limited information occupy the main area of ​​the scene, with only a small portion containing voxelized foreground data. Therefore, 3D sparse convolution can effectively avoid potential redundancy in the model, improving efficiency without sacrificing accuracy. Besides the redundancy in radar point cloud data, the structural design of the 3D sparse convolution model itself can also introduce redundancy. Typically, 3D sparse convolution uses downsampling to extract point cloud features, but downsampling may increase rather than decrease the number of non-empty voxels, undoubtedly increasing unnecessary computational costs in subsequent stages.

[0049] There are two commonly used types of sparse convolution: spatial submanifold sparse convolution and spatial regular sparse convolution. Considering that background regions typically constitute the majority of 3D scenes, directly performing submanifold convolution at various points on the input feature map inevitably involves a large amount of unnecessary computation, leading to computational redundancy. Therefore, to alleviate this problem, this application proposes Spatial Pruned Submanifold Sparse Convolution (SPSS-Conv) for dynamically checking regions of interest. The "expansion" effect of regular sparse convolution inevitably leads to the activation of a large number of adjacent locations, which imposes a greater computational burden on subsequent layers. The spatial dynamic sparsity in sparse convolution is crucial for complex 3D object detection.

[0050] This application can effectively reduce the computational cost of sparse convolution, enabling lightweight 3D object detection.

[0051] After introducing the application scenarios and design concepts of the embodiments of this application, the technical solutions provided by the embodiments of this application will be described below.

[0052] like Figure 1 As shown, this application provides a lightweight 3D target detection method based on sparse convolution pruning, including:

[0053] Step 101: Perform voxelization on the original point cloud data to obtain multiple voxels;

[0054] To process irregular point cloud data, it is necessary to convert the point cloud into regular voxels. First, given the point cloud data, the 3D space is subdivided into equally spaced cuboid voxels, with the corresponding size defined as V. D V H and V W The rectangular voxels generate a 3D voxel mesh of number N. D N H and N W The expressions for the three are as follows:

[0055]

[0056] Where D, H, and W represent the depth, height, and width of the original 3D point cloud data.

[0057] Step 102: Process multiple voxels using pre-trained convolutional blocks to obtain the first feature map;

[0058] like Figure 2 As shown, the first layer is the feature map obtained by convolution operation on the voxelized point cloud data. Since the input of the convolution is the point cloud data, it is impossible to determine the importance of the features. Therefore, this convolution is a normal 3D sparse convolution block, rather than a 3D sparse convolution pruning module.

[0059] Step 103: Process the first feature map using K pre-trained, sequentially connected downsampled sparse convolutional blocks to obtain K downsampled feature maps in sequence: the second feature map, the third feature map, ... and the (K+1)th feature map;

[0060] The downsampling sparse convolutional block includes a feature map partitioning module and a downsampling module; the feature map partitioning module is used to divide each cube with feature values ​​in the input 3D feature map into important cubes P. im and unimportant cube P nim Utilizing all important cubes P im Generate an important feature map of the same size as the input 3D feature map, utilizing all unimportant cubes P. nim The module generates an unimportant feature map of the same size as the input 3D feature map. The downsampling module consists of two parallel first sparse convolutional pruning layers and a downsampling processing unit, as well as a first concatenation unit. The first sparse convolutional pruning layer downsamples the important feature map to obtain a first downsampled feature map. The downsampling processing unit performs pooling on the feature map of the unimportant feature map to obtain a second downsampled feature map, which is the same size as the first downsampled feature map. The first concatenation unit concatenates the first and second downsampled feature maps together to generate the output 3D feature map.

[0061] In this process, each cube with eigenvalues ​​in the input 3D feature map is divided into importance cubes P. im and unimportant cube P nim ;include:

[0062] Calculate the mean of the features of all cubes in the input 3D feature map;

[0063] If a cube's feature is greater than the mean of all cubes' features, then that cube is considered an important cube, P. im Otherwise, the cube is an unimportant cube P. nim .

[0064] Utilizing all important cubes P im Generate important feature maps of the same size as the input 3D feature map, including:

[0065] Generate an empty feature map of the same size as the input 3D feature map, and include all important cubes P. im Replace the cubes at the corresponding positions in the empty feature map with the cubes at their original positions, thus creating the important feature map.

[0066] For the k-th downsampled sparse convolutional block, 1≤k≤K, the input of the k-th downsampled sparse convolutional block is of size (W k H k D k C k The k-th feature map of ) W k H k and D k Let C be the width, height, and depth of the k-th feature map. k Let W be the dimension of the k-th feature map; the output of the k-th downsampled sparse convolutional block is of size (W). k+1 H k+1 D k+1 C k+1 The (k+1)th feature map of ) W k+1 H k+1 and D k+1 Let C be the width, height, and depth of the (k+1)th feature map. k+1 Let be the dimension of the (k+1)th feature map; the number of convolutional kernels in the first sparse convolutional pruning layer of the k-th downsampled sparse convolutional block is n, the dimension of the convolutional kernels is w*w*d, the stride is s, and the padding value is p; then we have:

[0067]

[0068]

[0069]

[0070] C k+1 =n

[0071] Figure 2 This is a schematic diagram of three downsampled sparse convolutional blocks.

[0072] Step 104: Copy the (K+1)th feature map to obtain the (K+2)th feature map; process the (K+2)th feature map using K pre-trained sequentially connected upsampled sparse convolutional blocks to obtain K upsampled feature maps in sequence: the (K+3)th feature map, the (K+4)th feature map, ... and the 2Kth feature map;

[0073] The upsampling sparse convolutional block includes a feature map partitioning module and an upsampling module; the feature map partitioning module is used to divide each cube with feature values ​​in the input 3D feature map into important cubes P. im and unimportant cube P nim Utilizing all important cubes P im Generate an important feature map of the same size as the input 3D feature map, utilizing all unimportant cubes P. nim The module generates an unimportant feature map of the same size as the input 3D feature map. The upsampling module consists of two parallel second sparse convolutional pruning layers, an upsampling processing unit, and a second concatenation unit. The second sparse convolutional pruning layer upsamples the important feature map to obtain a first upsampled feature map. The upsampling processing unit interpolates the unimportant feature map to obtain a second upsampled feature map, which is the same size as the first upsampled feature map. The second concatenation unit concatenates the first and second upsampled feature maps to generate the output 3D feature map.

[0074] For the k-th upsampled sparse convolutional block, K+2≤k≤2K, the input of the k-th upsampled sparse convolutional block is of size (W k H k D k C k The k-th feature map of ) W k H k and D k Let C be the width, height, and depth of the k-th feature map. k Let K be the dimension of the k-th feature map, K+2≤k≤2K; the output of the k-th downsampled sparse convolutional block is of size (W). k+1 H k+1 D k+1 C k+1 The (k+1)th feature map of ) W k+1 H k+1 and D k+1 Let C be the width, height, and depth of the (k+1)th feature map. k+1 Let be the dimension of the (k+1)th feature map; the number of convolutional kernels in the second sparse convolutional pruning layer of the kth upsampled sparse convolutional block is n, the dimension of the convolutional kernels is w*w*d, the stride is s, and the padding value is p, then we have:

[0075] W k+1 =(W k -1)*s-2*p+w

[0076] H k+1 =(H k -1)*s-2*p+w

[0077] Dk+1 =(D k -1)*s-2*p+d

[0078] C k+1 =n

[0079] Figure 2 This is a schematic diagram of three upsampled sparse convolutional blocks.

[0080] Step 105: Add each corresponding element of two feature maps of the same scale in the 2K feature maps to obtain K final feature maps; use K pre-trained detection heads of different scales to process the K final feature maps to obtain the 3D target detection result.

[0081] By using different detector heads with the extracted features at different scales, receptive fields of different scales can be obtained, thereby enabling the detection of point cloud features at different scales. For example... Figure 2 As shown, the first and eighth feature maps are the same size, the second and seventh feature maps are the same size, the third and sixth feature maps are the same size, and the fourth and fifth feature maps are the same size. Detection is performed using four different sizes of detection heads.

[0082] In addition, the method also includes the step of jointly training the convolutional block, K downsampled sparse convolutional blocks, K upsampled sparse convolutional blocks, and K detection heads.

[0083] Based on the above embodiments, this application provides a lightweight 3D target detection device based on sparse convolution pruning, see below. Figure 3 As shown, the lightweight 3D target detection device 200 based on sparse convolution pruning provided in this application embodiment includes at least:

[0084] The voxel processing unit 201 is used to perform voxelization processing on the raw point cloud data to obtain multiple voxels;

[0085] Processing unit 202 is used to process multiple voxels using pre-trained convolutional blocks to obtain a first feature map;

[0086] The downsampling unit 203 is used to process the first feature map using K sequentially connected downsampling sparse convolutional blocks that have been pre-trained, and to obtain K downsampling feature maps in sequence: the second feature map, the third feature map, ... and the (K+1)th feature map;

[0087] Upsampling unit 204 is used to copy the (K+1)th feature map to obtain the (K+2)th feature map; the (K+2)th feature map is processed by K sequentially connected upsampling sparse convolutional blocks that have been pre-trained to obtain K upsampling feature maps in sequence: the (K+3)th feature map, the (K+4)th feature map, ... and the 2Kth feature map;

[0088] The detection unit 205 is used to add each corresponding element of two feature maps of the same scale in the 2K feature maps to obtain K final feature maps; and to process the K final feature maps using K pre-trained detection heads of different scales to obtain the three-dimensional target detection result.

[0089] It should be noted that the principle of the lightweight 3D target detection device 200 based on sparse convolution pruning provided in this application embodiment to solve the technical problem is similar to the method provided in this application embodiment. Therefore, the implementation of the lightweight 3D target detection device 200 based on sparse convolution pruning provided in this application embodiment can refer to the implementation of the method provided in this application embodiment, and the repeated parts will not be described again.

[0090] Based on the above embodiments, this application also provides an electronic device, see below. Figure 4 As shown, the electronic device 300 provided in this application embodiment includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, it implements the lightweight 3D target detection method based on sparse convolution pruning provided in this application embodiment.

[0091] The electronic device 300 provided in this application embodiment may further include a bus 303 connecting different components (including processor 301 and memory 302). The bus 303 represents one or more types of bus structures, including memory bus, peripheral bus, local area bus, etc.

[0092] The memory 302 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 3021 and / or cache memory 3022, and may further include read-only memory (ROM) 3023.

[0093] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, including but not limited to: an operating subsystem, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0094] Electronic device 300 can also communicate with one or more external devices 304 (e.g., keyboard, remote control, etc.), and with one or more devices that enable a user to interact with electronic device 300 (e.g., mobile phone, computer, etc.), and / or with any device that enables electronic device 300 to communicate with one or more other electronic devices 300 (e.g., router, modem, etc.). This communication can be performed through input / output (I / O) interface 305. Furthermore, electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 306. Figure 4 As shown, network adapter 306 communicates with other modules of electronic device 300 via bus 303. It should be understood that, although... Figure 4 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) subsystems, tape drives, and data backup storage subsystems.

[0095] It should be noted that, Figure 4 The electronic device 300 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0096] This application also provides a computer-readable storage medium storing computer instructions. When executed by a processor, these instructions implement the lightweight 3D object detection method based on sparse convolution pruning provided in this application. Specifically, the executable program can be built into or installed in an electronic device 300, so that the electronic device 300 can implement the lightweight 3D object detection method based on sparse convolution pruning provided in this application by executing the built-in or installed executable program.

[0097] The lightweight 3D target detection method based on sparse convolution pruning provided in this application embodiment can also be implemented as a program product. The program product includes program code. When the program product can run on the electronic device 300, the program code is used to make the electronic device 300 execute the lightweight 3D target detection method based on sparse convolution pruning provided in this application embodiment.

[0098] The program product provided in this application embodiment can be any combination of one or more readable media, wherein the readable media can be a readable signal medium or a readable storage medium, and the readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0099] The program product provided in this application embodiment can be a CD-ROM and include program code, and can also run on a computing device. However, the program product provided in this application embodiment is not limited thereto. In this application embodiment, the readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0100] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0101] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of this application do not depart from the spirit and scope of the technical solutions of this application, and should all be covered within the scope of the claims of this application.

Claims

1. A lightweight three-dimensional target detection method based on sparse convolution pruning, characterized in that, include: The original point cloud data is voxelized to obtain multiple voxels; The first feature map is obtained by processing multiple voxels using pre-trained convolutional blocks. The first feature map is processed by using a pre-trained K sequentially connected down-sampling sparse convolution blocks to sequentially obtain K down-sampled feature maps: a second feature map, a third feature map, and a (K+1)th feature map. and a (K+1)th feature map. The (K+1)th feature map is copied to obtain the (K+2)th feature map. The (K+2)th feature map is then processed using K pre-trained, sequentially connected upsampled sparse convolutional blocks to obtain K upsampled feature maps: the (K+3)th feature map, the (K+4)th feature map, etc. and the 2K feature map; Add each corresponding element of two feature maps of the same scale from the 2K feature maps to obtain K final feature maps; The three-dimensional object detection results are obtained by processing the K final feature maps using K pre-trained detection heads of different scales. The downsampling sparse convolutional block includes a feature map partitioning module and a downsampling module; the feature map partitioning module is used to divide each cube with feature values ​​in the input 3D feature map into important cubes. and unimportant cubes Utilize all important cubes Generate an important feature map of the same size as the input 3D feature map, utilizing all unimportant cubes. The module generates an unimportant feature map of the same size as the input 3D feature map. The downsampling module consists of two parallel first sparse convolutional pruning layers and a downsampling processing unit, as well as a first concatenation unit. The first sparse convolutional pruning layer downsamples the important feature map to obtain a first downsampled feature map. The downsampling processing unit performs pooling on the feature map of the unimportant feature map to obtain a second downsampled feature map, which is the same size as the first downsampled feature map. The first concatenation unit concatenates the first and second downsampled feature maps together to generate the output 3D feature map. The input to the k-th downsampled sparse convolutional block is of size ... The k-th feature map, Let the width, height, and depth of the k-th feature map be denoted as . Let k be the dimension of the k-th feature map. The output of the k-th downsampled sparse convolutional block is of size [value missing]. The (k+1)th feature map, Let the width, height, and depth of the (k+1)th feature map be... Let n be the dimension of the (k+1)th feature map; the number of convolutional kernels in the first sparse convolutional pruning layer of the k-th downsampled sparse convolutional block is n, and the dimension of the convolutional kernels is n. The sliding step size is The fill value is Then we have: The upsampling sparse convolutional block includes a feature map partitioning module and an upsampling module; the feature map partitioning module is used to divide each cube with feature values ​​in the input 3D feature map into important cubes. and unimportant cubes Utilize all important cubes Generate an important feature map of the same size as the input 3D feature map, utilizing all unimportant cubes. The system generates an unimportant feature map of the same size as the input 3D feature map. The upsampling module consists of two parallel second sparse convolutional pruning layers, an upsampling processing unit, and a second concatenation unit. The second sparse convolutional pruning layer upsamples the important feature map to obtain a first upsampled feature map. The upsampling processing unit interpolates the unimportant feature map to obtain a second upsampled feature map, which is the same size as the first upsampled feature map. The second concatenation unit concatenates the first and second upsampled feature maps to generate the output 3D feature map. The input to the k-th upsampled sparse convolutional block is of size ... The k-th feature map, Let the width, height, and depth of the k-th feature map be denoted as . Let k be the dimension of the k-th feature map. The output of the k-th downsampled sparse convolutional block is of size [value missing]. The (k+1)th feature map, Let the width, height, and depth of the (k+1)th feature map be... Let n be the dimension of the (k+1)th feature map; the number of convolutional kernels in the second sparse convolutional pruning layer of the k-th upsampled sparse convolutional block is n, and the dimension of the convolutional kernels is n. The sliding step size is The fill value is Then we have: 。 2. The method of claim 1, wherein, The raw point cloud data is voxelized to obtain multiple voxels, including: The original 3D point cloud data is voxelized using a cuboid voxel with a preset size to generate a three-dimensional voxel : ， ， wherein, , and are three dimension values of the cuboid voxel; and are the depth, height and width of the original 3D point cloud data, , and are the number of voxels in three directions.

3. The method of claim 1, wherein, dividing each cube having a feature value of the input three-dimensional feature map into an important cube and an unimportant cube ; comprising: ; comprising: Calculate the mean of the features of all cubes in the input 3D feature map; determining whether the characteristic of each cube is greater than the mean of the characteristics of all the cubes, then the cube is an important cube otherwise, the cube is an unimportant cube .

4. The method of claim 1, wherein, The method further includes the step of jointly training the convolutional block, K downsampled sparse convolutional blocks, K upsampled sparse convolutional blocks, and K detection heads.

5. A lightweight three-dimensional target detection device based on sparse convolution pruning, characterized in that, include: The voxel processing unit is used to voxelize the raw point cloud data to obtain multiple voxels. The processing unit is used to process multiple voxels using pre-trained convolutional blocks to obtain the first feature map; a downsampling unit, configured to process the first feature map by using K sequentially connected downsampling sparse convolution blocks which are pre-trained, to sequentially obtain K down-sampled feature maps, i.e., a second feature map, a third feature map, and a (K+1)th feature map and a (K+1)th feature map. The upsampling unit is used to copy the (K+1)th feature map to obtain the (K+2)th feature map; the (K+2)th feature map is then processed using K pre-trained, sequentially concatenated upsampling sparse convolutional blocks to obtain K upsampling feature maps: the (K+3)th feature map, the (K+4)th feature map, etc. and the 2K feature map; The detection unit is used to add each corresponding element of two feature maps of the same scale in the 2K feature maps to obtain K final feature maps; The three-dimensional object detection results are obtained by processing the K final feature maps using K pre-trained detection heads of different scales. The downsampling sparse convolutional block includes a feature map partitioning module and a downsampling module; the feature map partitioning module is used to divide each cube with feature values ​​in the input 3D feature map into important cubes. and unimportant cubes Utilize all important cubes Generate an important feature map of the same size as the input 3D feature map, utilizing all unimportant cubes. The module generates an unimportant feature map of the same size as the input 3D feature map. The downsampling module consists of two parallel first sparse convolutional pruning layers and a downsampling processing unit, as well as a first concatenation unit. The first sparse convolutional pruning layer downsamples the important feature map to obtain a first downsampled feature map. The downsampling processing unit performs pooling on the feature map of the unimportant feature map to obtain a second downsampled feature map, which is the same size as the first downsampled feature map. The first concatenation unit concatenates the first and second downsampled feature maps together to generate the output 3D feature map. The input to the k-th downsampled sparse convolutional block is of size ... The k-th feature map, Let the width, height, and depth of the k-th feature map be denoted as . Let k be the dimension of the k-th feature map. The output of the k-th downsampled sparse convolutional block is of size [value missing]. The (k+1)th feature map, Let the width, height, and depth of the (k+1)th feature map be... Let n be the dimension of the (k+1)th feature map; the number of convolutional kernels in the first sparse convolutional pruning layer of the k-th downsampled sparse convolutional block is n, and the dimension of the convolutional kernels is n. The sliding step size is The fill value is Then we have: The upsampling sparse convolutional block includes a feature map partitioning module and an upsampling module; the feature map partitioning module is used to divide each cube with feature values ​​in the input 3D feature map into important cubes. and unimportant cubes Utilize all important cubes Generate an important feature map of the same size as the input 3D feature map, utilizing all unimportant cubes. The system generates an unimportant feature map of the same size as the input 3D feature map. The upsampling module consists of two parallel second sparse convolutional pruning layers, an upsampling processing unit, and a second concatenation unit. The second sparse convolutional pruning layer upsamples the important feature map to obtain a first upsampled feature map. The upsampling processing unit interpolates the unimportant feature map to obtain a second upsampled feature map, which is the same size as the first upsampled feature map. The second concatenation unit concatenates the first and second upsampled feature maps to generate the output 3D feature map. The input to the k-th upsampled sparse convolutional block is of size ... The k-th feature map, Let the width, height, and depth of the k-th feature map be denoted as . Let k be the dimension of the k-th feature map. The output of the k-th downsampled sparse convolutional block is of size [value missing]. The (k+1)th feature map, Let the width, height, and depth of the (k+1)th feature map be... Let n be the dimension of the (k+1)th feature map; the number of convolutional kernels in the second sparse convolutional pruning layer of the k-th upsampled sparse convolutional block is n, and the dimension of the convolutional kernels is n. The sliding step size is The fill value is Then we have: 。 6. An electronic device, comprising: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described in any one of claims 1-4.

Images