A point cloud detection method, apparatus, and electronic device without sparse convolution dependency

Abstract

This application provides a point cloud detection method, apparatus, and electronic device that does not rely on sparse convolution. The method includes: preprocessing a LiDAR point cloud to generate a sparse BEV feature map and a binary occupancy mask; inputting the sparse BEV feature map and the binary occupancy mask into a backbone network; processing the sparse BEV feature map using density-aware convolution in the first N levels of the backbone network; and extracting multi-scale BEV features using standard two-dimensional convolution in subsequent levels of the backbone network. The density-aware convolution process involves: multiplying the input feature map element-wise with the binary occupancy mask and performing standard two-dimensional convolution to obtain the original convolutional feature map; convolving the binary occupancy mask using an auxiliary convolution kernel and counting the number of effective pixels in the receptive field at each output position; density scaling the original convolutional feature map according to the ratio of the number of effective pixels to the area of ​​the convolution kernel; and obtaining the target detection result based on the output of the backbone network.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a point cloud detection method, apparatus, and electronic device that does not depend on sparse convolution. Background Technology

[0002] In the 3D point cloud detection method based on column coding, the LiDAR point cloud is first divided into vertical columns and features are extracted from each column. Then, the column features are mapped to the bird's-eye view through a scattering operation. Figure 2 A 3D grid is used to form a bird's-eye view feature map. Due to the natural sparsity of lidar point clouds, more than 95% of the positions in the scattered bird's-eye view feature map are zero values, meaning that the effective features are highly sparsely distributed in space.

[0003] To address the aforementioned sparsity, high-performance 3D point cloud detection methods typically employ sparse convolution operators, which significantly improve computational efficiency by skipping zero-value regions and calculating only valid pixels. However, the implementation of sparse convolution operators relies on dedicated computing libraries, involving custom kernel scheduling and index management mechanisms, which presents significant deployment limitations. Specifically, in edge platforms such as drones and robots, power consumption and weight constraints typically necessitate dedicated inference chips rather than general-purpose graphics processors. These edge inference chips only support standard operators and cannot support the custom kernel scheduling and index management required for sparse convolution. Therefore, existing solutions for sparse convolution still rely on custom sparse operators or dedicated hardware accelerators, making direct deployment on general-purpose edge chips impossible.

[0004] If standard 2D convolution is directly used instead of sparse convolution, the zero-value regions in the sparse feature map will contaminate the boundaries of effective features. Specifically, when the convolution kernel slides at the boundaries of effective pixels, its receptive field simultaneously covers both the effective region and the zero-value region, causing the effective features to be diluted by zero values, the quality of boundary features to decrease, and ultimately resulting in reduced detection accuracy.

[0005] In the field of image inpainting, there exists a partial convolution method that distinguishes between valid and invalid pixels during convolution and performs scaling compensation on the output after convolution. This method's scaling strategy involves multiplying the convolution output by the ratio of the total receptive field area to the number of valid pixels, aiming to amplify the response attenuation caused by incomplete valid pixels back to a normal level. However, this partial convolution method was originally designed for image inpainting tasks, where the invalid region typically accounts for 10% to 30%, resulting in a relatively mild amplification factor. In bird's-eye view feature maps, however, the invalid region accounts for over 95%, at which point the amplification factor can reach several to tens of times the convolution kernel area. For example, for a 3×3 convolution, the amplification factor can reach over 9 times, causing a sharp expansion of sparse region feature values ​​and disrupting the overall numerical distribution of the feature map. Therefore, this method cannot be directly applied to bird's-eye view scenarios. Since the partial convolution method was proposed in 2018, despite the long-standing problem of zero-value contamination in sparse feature maps in the field of bird's-eye view detection, there has been no publicly available solution that applies the scaling strategy of partial convolution to bird's-eye view point cloud detection. This indicates that the inapplicability of this method in extremely sparse scenarios has been recognized by the field.

[0006] In summary, among relevant 3D point cloud detection methods, sparse convolution operators are difficult to deploy on general-purpose edge inference chips, while directly using standard 2D convolution leads to zero-value contamination. Furthermore, some convolution methods exhibit numerical distribution instability in extremely sparse bird's-eye view scenarios. Therefore, there is an urgent need for a point cloud detection method entirely based on standard 2D convolution, capable of eliminating the contamination of effective features by zero values ​​in sparse feature maps, achieving equal or better detection accuracy without relying on sparse convolution, and enabling direct deployment on general-purpose edge inference chips. Summary of the Invention

[0007] In view of this, this application proposes a point cloud detection method, apparatus and electronic device that are independent of sparse convolution.

[0008] Specifically, this application is implemented through the following technical solution:

[0009] According to a first aspect of the embodiments of this specification, a point cloud detection method without sparse convolution dependency is provided, comprising the following steps:

[0010] Step S1: Perform cylinder encoding on the lidar point cloud, generate a sparse BEV feature map through scattering operation, and generate a binary occupancy mask indicating the position of non-empty cylinders.

[0011] Step S2: Input the sparse BEV feature map and the binary occupancy mask into the backbone network. In the first N levels of the backbone network, density-aware convolution is used to process the sparse BEV feature map. In the subsequent levels of the backbone network, standard two-dimensional convolution is used to extract multi-scale BEV features. The density-aware convolution process is as follows:

[0012] After multiplying the input feature map element-wise with the binary occupancy mask, a standard two-dimensional convolution is performed to obtain the original convolutional feature map.

[0013] The binary occupancy mask is convolved using an auxiliary convolution kernel, and the number of effective pixels in the receptive field at each output position is counted.

[0014] The original convolutional feature map is density scaled according to the ratio of the number of effective pixels to the area of ​​the convolutional kernel to obtain a density-scaled feature map, and the binary occupancy mask is updated.

[0015] Step S3: Obtain the target detection result of the lidar point cloud based on the output of the backbone network.

[0016] According to a second aspect of the embodiments of this specification, a point cloud detection apparatus without sparse convolution dependency is provided, comprising:

[0017] The point cloud processing unit is used to encode the LiDAR point cloud into cylinders, generate a sparse BEV feature map through scattering operations, and generate a binary occupancy mask indicating the location of non-empty cylinders.

[0018] The feature map processing unit is used to input the sparse BEV feature map and the binary occupancy mask into the backbone network. In the first N levels of the backbone network, the sparse BEV feature map is processed by density-aware convolution. In the subsequent levels of the backbone network, multi-scale BEV features are extracted using standard two-dimensional convolution. The density-aware convolution process is as follows:

[0019] After multiplying the input feature map element-wise with the binary occupancy mask, a standard two-dimensional convolution is performed to obtain the original convolutional feature map.

[0020] The binary occupancy mask is convolved using an auxiliary convolution kernel, and the number of effective pixels in the receptive field at each output position is counted.

[0021] The original convolutional feature map is density scaled according to the ratio of the number of effective pixels to the area of ​​the convolutional kernel to obtain a density-scaled feature map, and the binary occupancy mask is updated.

[0022] The target detection unit is used to obtain the target detection result of the lidar point cloud based on the output of the backbone network.

[0023] According to a third aspect of the embodiments of this specification, an electronic device is provided, comprising:

[0024] processor;

[0025] A computer-readable storage medium storing computer program instructions that, when executed by the processor, cause the processor to perform the method as described in the first aspect.

[0026] The embodiments of this application have at least the following technical effects:

[0027] First, the entire detection process in this application embodiment only uses standard two-dimensional convolution, batch normalization, activation functions and other common operators, which can be directly exported as a standard model exchange format and deployed to edge inference chips that only support standard operators, thus having the widest chip compatibility.

[0028] Second, since their inception, some convolutional methods have long suffered from sparse zero-value contamination in the field of bird's-eye view detection. The embodiments of this application reveal that the root cause lies in the direction of their magnification compensation (…). Under extremely sparse conditions where over 95% is zero, a magnification factor of several to tens of times is generated, leading to a collapse of the numerical distribution; based on this, the embodiments of this application reverse the scaling direction to the occupancy density ratio ( The scaling factor is always in the range of 0 to 1, and the feature response naturally reflects the local point cloud density. The ablation experiment shows that the density scaling direction has an average accuracy 0.84 percentage points higher than the magnification compensation direction.

[0029] Third, in the evaluation of the public test set, the embodiments of this application outperformed the baseline methods that rely on sparse convolution in both the mean accuracy and the detection score, proving that a pipeline based entirely on standard convolution can achieve and surpass the detection accuracy of sparse convolution. Attached Figure Description

[0030] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Some specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings in an exemplary and non-limiting manner. The same reference numerals in the drawings indicate the same or similar parts or components. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0031] Figure 1 This is a schematic diagram illustrating an exemplary embodiment of a point cloud detection method without sparse convolution dependency.

[0032] Figure 2This is a schematic diagram illustrating the overall processing flow of a lidar point cloud according to an exemplary embodiment of this application;

[0033] Figure 3 This is a schematic diagram illustrating the processing of a density-aware convolutional structure according to an exemplary embodiment of this application;

[0034] Figure 4 This is a structural block diagram of an electronic device illustrated in an exemplary embodiment of this application;

[0035] Figure 5 This is a structural block diagram of a point cloud detection device without sparse convolution dependency, as illustrated in an exemplary embodiment of this application. Detailed Implementation

[0036] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0037] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0038] To address the issues that existing point cloud detection methods rely on sparse convolution, making them unsuitable for deployment on edge chips, or that direct substitution leads to decreased accuracy, this application proposes a point cloud detection scheme for edge chip deployment without sparse convolution dependency. This scheme eliminates the contamination of effective features by zero values ​​in the sparse bird's-eye view feature map through density-aware convolution. The entire chain uses only standard operators and can be directly deployed on edge inference chips that only support standard operators.

[0039] The embodiments described in this specification will now be described in detail.

[0040] This application provides a point cloud detection method without sparse convolution dependency. Figure 1 This is a schematic diagram illustrating an exemplary embodiment of a point cloud detection method without sparse convolution dependency, as shown in this application. Figure 1 As shown, the point cloud detection method includes at least the following steps:

[0041] Step S1: Perform column encoding on the lidar point cloud, generate a sparse BEV (Bird's-EyeView) feature map through scattering operation, and generate a binary occupancy mask indicating the location of non-empty columns.

[0042] Step S2: Input the sparse BEV feature map and the binary occupancy mask into the backbone network. In the first N levels of the backbone network, density-aware convolution is used to process the sparse BEV feature map. In the subsequent levels of the backbone network, standard two-dimensional convolution is used to extract multi-scale BEV features. The density-aware convolution process is as follows:

[0043] After multiplying the input feature map element-wise with the binary occupancy mask, a standard two-dimensional convolution is performed to obtain the original convolutional feature map.

[0044] The binary occupancy mask is convolved using an auxiliary convolution kernel, and the number of effective pixels in the receptive field at each output position is counted.

[0045] The original convolutional feature map is density-scaled according to the ratio of the number of effective pixels to the area of ​​the convolutional kernel to obtain a density-scaled feature map, and the binary occupancy mask is updated.

[0046] Step S3: Obtain the target detection result of the lidar point cloud based on the output of the backbone network.

[0047] The convolution operations used in steps S1 to S3 above are all standard two-dimensional convolutions, without sparse convolution operators or custom computation kernels.

[0048] It is worth noting that the standard two-dimensional convolution described in this application embodiment refers to the convolution kernel performing sliding calculations at all spatial locations of the input feature map without skipping or selectively calculating any location. Its calculation process does not rely on sparse index management, custom kernel scheduling, or dedicated sparse computation libraries. It is a regular convolution operation natively supported by deep learning frameworks (such as PyTorch and TensorFlow) and is also a standard operator natively supported by the instruction set of edge inference chips.

[0049] In some embodiments, the auxiliary convolutional kernel in step S2 has the same kernel size, stride, and padding parameters as the main convolutional kernel of the backbone network.

[0050] In some embodiments, the weights of the auxiliary convolutional kernel in step S2 are constants of all 1s.

[0051] In some embodiments, the binary occupancy mask is updated in step S2 by the following steps:

[0052] After the binary occupancy mask is convolved by the auxiliary convolution kernel, the positions with output values ​​greater than zero are updated to be valid, so that the valid region spreads to the surrounding areas layer by layer with the network depth.

[0053] In some embodiments, step S3 includes:

[0054] The feature maps at various levels output by the backbone network are fused at multiple scales using feature pyramids to obtain the fused bird's-eye view features.

[0055] The detection head is used to perform target detection on the fused bird's-eye view features, and the category, 3D bounding box and confidence score of each target are obtained.

[0056] In some embodiments, the method further includes the following steps:

[0057] Step S4: The batch normalization parameters in the target detection model using the backbone network are folded into the weights and biases of the corresponding convolutional layers, and the folded model is exported as a model exchange format containing only standard operators. After quantization, it is compiled and deployed to the edge inference chip.

[0058] The following is combined Figure 2 and Figure 3 The point cloud detection process of this application is described in detail.

[0059] like Figure 1 As shown, in this embodiment of the application, step S1 is first performed to encode and scatter the cylinder.

[0060] The raw point cloud acquired by LiDAR is encoded into cylinders, dividing the 3D space into vertical cylinders. Point clouds falling into the same cylinder are aggregated into a feature vector for that cylinder through a dynamic feature extraction network. Then, the cylinder features are mapped to a bird's-eye view through a scattering operation. Figure 2 The corresponding positions of the 3D grid are used to generate sparse BEV feature maps. Due to the inherent sparsity of LiDAR point clouds, more than 95% of the positions in the scattered BEV feature maps are zero values.

[0061] Simultaneously, a binary occupancy mask is generated to mark the locations of non-empty pillars in the sparse BEV feature map as valid, and the remaining locations as invalid. For example, a valid value is 1, and an invalid value is 0.

[0062] Then, step S2 is performed, which is a multi-scale feature extraction process. The multi-scale feature extraction process includes a feature extraction process based on density-aware convolution and a feature extraction process based on standard two-dimensional convolution.

[0063] like Figure 2 In the first N levels of the backbone network, exemplarily N=2, density-aware convolution is used to process the sparse BEV feature maps to eliminate the contamination of effective features by zero-value regions.

[0064] like Figure 3 As shown, the feature extraction process based on density-aware convolution is as follows:

[0065] Main convolution step: The input feature map is multiplied element-wise with the occupancy mask to ensure that only the effective regions participate in the calculation. Then, the multiplication result is subjected to standard two-dimensional convolution to obtain the original convolution feature map.

[0066] Effective pixel counting stage: An auxiliary convolutional kernel with fixed weights of all 1s, which is not involved in training, is used to convolve the binary occupancy mask to count the number of effective pixels in the receptive field at each output position. The weights of this auxiliary convolutional kernel are constant during inference and can be optimized by the compiler.

[0067] Density scaling step: The original convolutional feature map is scaled according to the following formula:

[0068]

[0069] In formula (1), This represents the feature map after density scaling. Represents the original convolutional feature map. This indicates the number of effective pixels within the receptive field. This represents the area of ​​the convolution kernel. Taking a 3x3 convolution as an example, Scaling factor The occupancy density ratio is always within the range of 0 to 1.

[0070] In dense regions, the scaling factor is close to 1, and the feature retains its original strength; while in sparse regions, the scaling factor is close to 0, and the feature response decays proportionally to the occupied density.

[0071] Traditional partial convolution schemes employ a reverse compensation strategy, multiplying the convolution output by the ratio of the receptive field area to the effective number of pixels (i.e., In sparse regions, a scaling factor greater than 1 is generated to compensate for the attenuation caused by zero values. However, under extremely sparse conditions in the BEV feature map, this scaling factor can reach several times the area of ​​the convolution kernel (e.g., a 9-fold scaling factor in a 3x3 convolution), causing a sharp expansion of feature values ​​in sparse regions and disrupting the numerical distribution of the feature map. In the embodiments of this application, the density scaling factor is always less than or equal to 1, resulting in numerical stability.

[0072] This embodiment verifies numerical stability through experimental testing: The activation value distribution of the feature map at the first-level output of the backbone network was statistically analyzed. The mean activation value in the amplification compensation direction was 2.28, the standard deviation was 7.09, and the maximum value reached 198, with 1253 outliers exceeding 50 (0.32%). The mean activation value in the density scaling direction was 0.25, the standard deviation was 0.43, and the maximum value was only 8.2, with no values ​​exceeding 50. The standard deviation in the amplification compensation direction was 16 times that of the density scaling direction, and the maximum value was 24 times, confirming that under extremely sparse conditions of the BEV feature map, the amplification compensation strategy produces severe numerical inflation, while the density scaling strategy maintains a stable numerical distribution.

[0073] The aforementioned differences in numerical distribution have two impacts on subsequent processing: First, the batch normalization layer calculates the mean and variance per channel. Extremely large values ​​at a few locations amplify the channel variance, causing normal features to be compressed into an extremely narrow numerical range after normalization, reducing feature discriminative power. Second, edge deployment requires INT8 quantization. The quantization process linearly maps the floating-point value range to the integer range of -128 to 127. The value range in the amplification compensation direction is 0 to 198, while the density scaling direction is only 0 to 8.2. The effective features of the former are concentrated in the low-value range of 0 to 3, accounting for only 1.5% of the overall quantization range. Most quantization levels are wasted on extreme values, resulting in a severe loss of accuracy for the effective features after quantization. The compact numerical distribution in the density scaling direction allows for full utilization of the quantization levels, making it more suitable for low-precision inference on edge chips.

[0074] Mask update stage: After convolving the binary occupancy mask with an auxiliary convolution kernel, positions with output values ​​greater than zero are marked as valid. That is, as long as there is one valid pixel in the receptive field, the output position is marked as valid. As the network deepens, the valid region gradually spreads from the non-empty cylinder to the surrounding area.

[0075] It's worth noting that in this embodiment, density-aware convolution is only used in the first N stages of the backbone network, where the feature map is sparsest and the density scaling benefit is greatest. In subsequent stages, the effective region coverage is significantly increased due to the diffusion of the binary occupancy mask, and the density scaling factor approaches 1, degenerating into regular convolution. The entire module only uses three standard operators: standard 2D convolution, element-wise multiplication, and element-wise division, without requiring a custom computation kernel.

[0076] Finally, step S3 is performed: multi-scale feature fusion and feature detection.

[0077] After extracting multi-scale bird's-eye view features using a backbone network with a multi-scale residual structure, the backbone network is assumed to have five levels with channel numbers of 32, 64, 128, 256, and 256 respectively, and strides of 1, 2, 4, 8, and 16 respectively. The first two levels use density-aware convolutions, and the last three levels use standard 2D convolutions. All convolutions are standard 2D convolutions with batch normalization and activation functions. Residual connections are aligned using 1x1 convolutions when there is a channel mismatch.

[0078] The five-level feature maps are fed into the feature pyramid for multi-scale fusion. Through upsampling, stitching and convolution operations, a bird's-eye view feature map with uniform resolution is output for use by the detection head.

[0079] The entire backbone network and feature pyramid do not contain any non-standard operators and can be exported to a standard model exchange format with one click.

[0080] The detection head based on the center point performs target detection on the fused bird's-eye view features, locates the target center by the peak of the heat map, and regresses the three-dimensional bounding box size, orientation, velocity and other attributes of each target, and outputs the final detection result.

[0081] This embodiment also underwent experimental verification.

[0082] Systematic experimental validation was conducted on a large-scale autonomous driving point cloud dataset containing 10 target categories. The evaluation metrics were mean accuracy (mAP) and detection score (NDS). The training hardware consisted of two GPUs and was trained for 20 rounds.

[0083] Table 1 shows the scaling direction comparison ablation experiment. Under the same backbone network, detector head, and training configuration, only the scaling direction of the first two stages was changed:

[0084]

[0085] Table 1

[0086] Both schemes use the exact same network structure, training parameters, and data augmentation; the only difference is the direction of the scaling formula. Based on Table 1, it can be seen that under extremely sparse conditions, the amplification compensation strategy leads to a decrease in accuracy due to numerical inflation, while the density scaling strategy achieves significantly better detection performance by allowing the feature response to naturally reflect the occupied density.

[0087] This embodiment also performs object detection on the same public test set based on a pure two-dimensional convolutional baseline model, a sparse convolutional baseline model, and the model of this application embodiment, and submits the detection results to the evaluation server of the public test set for evaluation. The evaluation results are shown in Table 2:

[0088]

[0089] Table 2

[0090] Based on the evaluation results in Table 2, it can be seen that on this publicly available test set, the mAP and NDS of the embodiments in this application are the highest among the three, exceeding the sparse convolution baseline (mAP is 0.24 percentage points higher, and NDS is 0.08 percentage points higher). The NDS of the pure 2D convolution baseline is close to that of sparse convolution (the difference is only 0.12 percentage points), and it is further surpassed after adding density-aware convolution. It can be seen that the embodiments in this application are also superior to the sparse convolution baseline in terms of orientation error and attribute error metrics.

[0091] This embodiment compares with major point cloud detection methods on the same dataset, and the comparison results are shown in Table 3:

[0092]

[0093] Table 3

[0094] Based on the comparison results in Table 3, it can be seen that among the lightweight methods that do not require sparse convolution, the mAP and NDS of the embodiments of this application both outperform CenterPoint and MGTANet-P, which rely on sparse convolution. It is the only method that surpasses the sparse convolution baseline in both metrics and can be directly deployed to edge inference chips.

[0095] To verify the feasibility of deploying this solution on an edge inference chip, the trained model was quantized with INT8, compiled, and deployed to the Horizon Robotics RDK S100 edge computing platform (BPU Nash architecture, 80 TOPS). During model compilation, all operators were successfully mapped to standard operators supported by the BPU, requiring no custom operator adaptation. In pipelined inference mode, the end-to-end inference latency per frame was 52 milliseconds, and the inference frame rate was 19.2 frames per second, exceeding the data acquisition frame rate of typical LiDAR (10 to 20 frames per second), meeting real-time processing requirements.

[0096] Based on the above embodiments of this application, the point cloud detection solution of this application has at least the following advantages:

[0097] (1) No sparse convolution dependency in the entire pipeline: The entire detection pipeline only uses standard two-dimensional convolution, batch normalization, activation functions and other general operators, which can be directly exported as a standard model exchange format and deployed to edge inference chips that only support standard operators, with the widest chip compatibility.

[0098] (2) In the embodiments of this application, the scaling direction is reversed to the density ratio ( The scaling factor is always in the range of 0 to 1, and the feature response naturally reflects the local point cloud density. The ablation experiment shows that the density scaling direction has an average accuracy 0.84 percentage points higher than the magnification compensation direction.

[0099] (3) In the official test set evaluation, the embodiments of this application outperformed the baseline methods that rely on sparse convolution in both the mean accuracy and the detection score, proving that the pipeline based entirely on standard convolution can achieve and exceed the detection accuracy of sparse convolution.

[0100] (4) After INT8 quantization, it was successfully compiled and deployed to the edge inference chip (Horizon RDK S100, BPU Nash architecture, 80TOPS). All operators were successfully mapped to the standard operators supported by BPU without any custom operator adaptation. The measured inference frame rate was 19.2 frames per second, which met the real-time processing requirements.

[0101] Figure 4 This is a schematic diagram of an electronic device illustrated in this specification according to an exemplary embodiment. Please refer to... Figure 4 At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, memory 408, a hardware acceleration device 410, and non-volatile memory 412, and may also include other hardware required for its functions. One or more embodiments of this application can be implemented in software, for example, the processor 402 reads the corresponding computer program from the non-volatile memory 412 into memory 408 and then runs it. Of course, in addition to software implementation, one or more embodiments of this application do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the above processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0102] Figure 5 This is a structural block diagram illustrating a point cloud detection device without sparse convolution dependency, as shown in an exemplary embodiment of this application. The point cloud detection device can be applied to applications such as... Figure 4 The electronic device shown implements the technical solution of this application. The point cloud detection device includes: a point cloud processing unit 510, a feature map processing unit 520, and a target detection unit 530, wherein:

[0103] The point cloud processing unit 510 is used to encode the LiDAR point cloud into cylinders, generate a sparse BEV feature map through scattering operations, and generate a binary occupancy mask indicating the position of non-empty cylinders.

[0104] The feature map processing unit 520 is used to input the sparse BEV feature map and the binary occupancy mask into the backbone network. In the first N levels of the backbone network, the sparse BEV feature map is processed by density-aware convolution. In the subsequent levels of the backbone network, multi-scale BEV features are extracted using standard two-dimensional convolution. The density-aware convolution process is as follows:

[0105] After multiplying the input feature map element-wise with the binary occupancy mask, a standard two-dimensional convolution is performed to obtain the original convolutional feature map.

[0106] The binary occupancy mask is convolved using an auxiliary convolution kernel, and the number of effective pixels in the receptive field at each output position is counted.

[0107] The original convolutional feature map is density scaled according to the ratio of the number of effective pixels to the area of ​​the convolutional kernel to obtain a density-scaled feature map, and the binary occupancy mask is updated.

[0108] The target detection unit 530 is used to obtain the target detection result of the lidar point cloud based on the output of the backbone network.

[0109] In some embodiments, the target detection unit 530 is used to perform multi-scale fusion of the feature maps at all levels output by the backbone network through the feature pyramid to obtain the fused bird's-eye view features; and to use the detection head to perform target detection on the fused bird's-eye view features to obtain the category, three-dimensional bounding box and confidence score of each target.

[0110] In some embodiments, the point cloud detection device further includes a post-processing unit, which is used to fold the batch normalization parameters in the target detection model using the backbone network into the weights and biases of the corresponding convolutional layer, and export the folded model as a model exchange format containing only standard operators, and then compile and deploy it to the edge inference chip after quantization.

[0111] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0112] Accordingly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the above embodiments.

[0113] Accordingly, embodiments of this application also provide a computer program product configured to perform the methods described in any of the above embodiments.

[0114] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0115] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0116] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0117] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0118] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0119] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0120] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0121] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0122] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A point cloud detection method without sparse convolution dependency, characterized in that, Includes the following steps: Step S1: Perform cylinder encoding on the lidar point cloud, generate a sparse BEV feature map through scattering operation, and generate a binary occupancy mask indicating the position of non-empty cylinders. Step S2: Input the sparse BEV feature map and the binary occupancy mask into the backbone network. In the first N levels of the backbone network, density-aware convolution is used to process the sparse BEV feature map. In the subsequent levels of the backbone network, standard two-dimensional convolution is used to extract multi-scale BEV features. The density-aware convolution process is as follows: After multiplying the input feature map element-wise with the binary occupancy mask, a standard two-dimensional convolution is performed to obtain the original convolutional feature map. The binary occupancy mask is convolved using an auxiliary convolution kernel, and the number of effective pixels in the receptive field at each output position is counted. The original convolutional feature map is density scaled according to the ratio of the number of effective pixels to the area of ​​the convolutional kernel to obtain a density-scaled feature map, and the binary occupancy mask is updated. Step S3: Obtain the target detection result of the lidar point cloud based on the output of the backbone network.

2. The method according to claim 1, characterized in that, The auxiliary convolutional kernel in step S2 has the same kernel size, stride, and padding parameters as the main convolutional kernel of the backbone network.

3. The method according to claim 1, characterized in that, The weights of the auxiliary convolution kernel in step S2 are all constants of 1.

4. The method according to claim 1, characterized in that, The density-scaled feature map in step S2 is represented by the following formula: ； in, This is a density-scaled feature map. This is the original convolutional feature map. For the number of effective pixels, This represents the area of ​​the convolution kernel.

5. The method according to claim 2, characterized in that, In step S2, the binary occupancy mask is updated through the following steps: After the binary occupancy mask is convolved by the auxiliary convolution kernel, the positions with output values ​​greater than zero are updated to be valid, so that the valid region spreads to the surrounding areas layer by layer with the network depth.

6. The method according to claim 1, characterized in that, Step S3 includes: The feature maps at various levels output by the backbone network are fused at multiple scales using feature pyramids to obtain the fused bird's-eye view features. The detection head is used to perform target detection on the fused bird's-eye view features, and the category, 3D bounding box and confidence score of each target are obtained.

7. The method according to claim 1, characterized in that, The convolution operations used in steps S1 to S3 are all standard two-dimensional convolutions.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes the following steps: Step S4: The batch normalization parameters in the target detection model using the backbone network are folded into the weights and biases of the corresponding convolutional layers, and the folded model is exported as a model exchange format containing only standard operators. After quantization, it is compiled and deployed to the edge inference chip.

9. A point cloud detection device without sparse convolution dependency, characterized in that, include: The point cloud processing unit is used to encode the LiDAR point cloud into cylinders, generate a sparse BEV feature map through scattering operations, and generate a binary occupancy mask indicating the location of non-empty cylinders. The feature map processing unit is used to input the sparse BEV feature map and the binary occupancy mask into the backbone network. In the first N levels of the backbone network, the sparse BEV feature map is processed by density-aware convolution. In the subsequent levels of the backbone network, multi-scale BEV features are extracted using standard two-dimensional convolution. The density-aware convolution process is as follows: After multiplying the input feature map element-wise with the binary occupancy mask, a standard two-dimensional convolution is performed to obtain the original convolutional feature map. The binary occupancy mask is convolved using an auxiliary convolution kernel, and the number of effective pixels in the receptive field at each output position is counted. The original convolutional feature map is density scaled according to the ratio of the number of effective pixels to the area of ​​the convolutional kernel to obtain a density-scaled feature map, and the binary occupancy mask is updated. The target detection unit is used to obtain the target detection result of the lidar point cloud based on the output of the backbone network.

10. An electronic device, characterized in that, include: processor; A computer-readable storage medium storing computer program instructions that, when executed by the processor, cause the processor to perform the method as described in any one of claims 1 to 8.

Images