Laser multi-task sensing method and system for low-computing-power platform

Abstract

The invention relates to a laser multi-task sensing method and system for a low-computing-power platform. In the technical scheme of the invention, firstly, an original point cloud is downsampled by using a point cloud random sampling algorithm, and then the sampled point cloud is input into a multi-task perception model to obtain a 3D target detection result and a coarse-grained BEV grid multi-label semantic segmentation result. And finally, the coarse-grained semantic information is refined through a multi-label post-processing module, and a ground / drivable space extraction task, a road edge detection task and a non-standard obstacle detection task are completed.

Description

Technical Field

[0001] The present invention relates to the field of computer perception, and in particular to a laser multi-task perception method and system for a low computing power platform. Background Art

[0002] In the scenario of autonomous driving, Voxel, Pillar, and Range View are several commonly used point cloud expressions for LiDAR perception models when they are deployed. Voxel-based models often involve 3D convolution and large input and output feature quantities, which consume huge computing resources and computing time. They cannot be efficiently deployed on low-cost, low-computing platforms and meet the needs of real-time perception of autonomous driving. Pillar-based models consume relatively few computing resources and can meet the deployment requirements of low-computing platforms. However, for point cloud semantic segmentation tasks, the uniform division in height makes it impossible to distinguish different semantic types of point clouds within a single Pillar. Range View reconstructs point clouds into dense pseudo-images based on their scanning characteristics, can use 2D convolution for efficient reasoning, and can perform fine-grained semantic segmentation on each point, but compared with BEV, Range View has poor performance in 3D target detection tasks. Voxel and Pillar can both be directly compressed to BEV feature maps. BEV is a commonly used spatial expression in autonomous driving and is more suitable for multi-sensor fusion and feature time series fusion than Range View.

[0003] For low-computing deployment platforms with limited computing resources, it has become a trend to merge multiple tasks into a single network model and share part of the network structure. However, for point cloud semantic segmentation and point cloud 3D object detection tasks, semantic segmentation tasks require more refined spatial division (the amount of computation increases significantly as the granularity of spatial division becomes smaller), while 3D object detection tasks do not require such fine-grained spatial expression. Therefore, for autonomous driving use scenarios, it is necessary to better balance the amount of computation and computing task performance.

[0004] Therefore, there is a need for methods and systems that can improve upon the deficiencies in the prior art. Summary of the invention

[0005] This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0006] To address the above problems, the present invention uses a multi-task shared network structure and multi-label point cloud BEV semantic segmentation results to balance the multi-task performance and the computational efficiency of the entire system.

[0007] In the technical solution of the present invention, the original point cloud is first downsampled using a point cloud random sampling algorithm, and then the sampled point cloud is input into a multi-task perception model to obtain 3D target detection results and coarse-grained BEV grid multi-label semantic segmentation results. Finally, the coarse-grained semantic information is refined through a multi-label post-processing module to complete the ground / drivable space extraction task, curb detection task, and non-standard obstacle detection task.

[0008] Specifically, in one embodiment of the present invention, a method for laser multi-task perception for a low computing power platform is disclosed, the method comprising:

[0009] Receiving an original point cloud and performing point cloud downsampling on the original point cloud;

[0010] Performing BEV encoding on the downsampled point cloud to generate a BEV feature map;

[0011] Downsampling the BEV feature map to generate a downsampled BEV feature map;

[0012] performing a first feature decoding operation on the downsampled BEV feature map to generate a first feature map;

[0013] Performing BEV multi-label semantic segmentation on the first feature map to output a multi-label BEV semantic grid result;

[0014] performing a second feature decoding operation on the downsampled BEV feature map to generate a second feature map; and

[0015] 3D object detection is performed on the second feature map to output a 3D object detection box.

[0016] In one embodiment of the present invention, downsampling the original point cloud further includes:

[0017] Downsampling the point cloud inside each Pillar of the original point cloud; and

[0018] When the point cloud inside the Pillar exceeds the threshold, a local randomization operation is performed, where:

[0019] The threshold is the number of points N after downsampling in the Pillar multiplied by the expansion coefficient k. The setting of the expansion coefficient k is adjusted according to the randomness of the original point cloud data, and

[0020] The local randomization operation starts from the N+1th point and randomly replaces any point among the N points with a set probability p.

[0021] In one embodiment of the present invention, the BEV encoding includes compressing the point cloud input features within a Pillar under BEV from a 3D space to a 2D BEV space.

[0022] In one embodiment of the present invention, feature downsampling the BEV feature map further includes controlling the step size of the convolution layer so as to downsample the BEV feature map by a specific multiple, and the specific multiple is based on the step size.

[0023] In one embodiment of the present invention, the first feature decoding operation includes upsampling the downsampled BEV feature map, whereby the first feature map includes the upsampled feature map, and the second feature decoding operation includes pooling, upsampling and splicing the downsampled BEV feature map, whereby the second feature map includes the spliced ​​feature map, wherein the first feature map has a finer granularity than the second feature map.

[0024] In one embodiment of the present invention, performing BEV multi-label semantic segmentation on the first feature map further includes using a multi-label semantic segmentation loss function to supervise the output of the point cloud semantic segmentation task during the model training phase while keeping the single-label network structure unchanged.

[0025] In one embodiment of the present invention, the method further includes performing multi-label post-processing on the multi-label BEV semantic mesh result to complete subsequent tasks that require finer-grained point cloud semantic segmentation, wherein the subsequent tasks include curb detection tasks, non-standard obstacle detection tasks, and ground / drivable space extraction tasks.

[0026] In the above embodiment of the present invention, the multi-label post-processing includes performing refinement processing based on the multi-label BEV semantic grid result and point space information, wherein:

[0027] The curb detection task includes combining the grid with the curb and the height jump information of the curb to perform curb linear extraction to output a fine curb.

[0028] The non-standard obstacle detection task includes combining obstacle semantics and the spatial height of the point to output the obstacle feature point semantics, wherein when there are multiple obstacle semantics in the grid, clustering information and adjacent grid semantics are combined to distinguish them, and

[0029] The ground / drivable space extraction task includes ground fitting based on the semantic category of ground elements and the height of the lowest point in the Pillar, and then outputting the drivable space based on the obstacle feature points and ground extraction results.

[0030] In the above embodiment of the present invention, completing the subsequent task further includes performing small batch staggered training of heterogeneous label data, including:

[0031] Divide the training data into data with 3D object detection annotations, data with point cloud semantic segmentation annotations, and data with both annotations; and

[0032] The three types of labeled data are polled in small batches with a batch as the smallest unit until all data in the data group with the largest data volume is traversed.

[0033] In another embodiment of the present invention, a system for laser multi-task perception for a low computing power platform is disclosed, the system comprising:

[0034] A point cloud downsampling module is configured to receive an original point cloud and perform point cloud downsampling on the original point cloud;

[0035] The BEV encoding module is configured to:

[0036] Performing BEV encoding on the downsampled point cloud to generate a BEV feature map; and

[0037] Downsampling the BEV feature map to generate a downsampled BEV feature map;

[0038] A feature decoding module, which is configured to:

[0039] performing a first feature decoding operation on the downsampled BEV feature map to generate a first feature map;

[0040] Performing BEV multi-label semantic segmentation on the first feature map to output a multi-label BEV semantic grid result;

[0041] performing a second feature decoding operation on the downsampled BEV feature map to generate a second feature map;

[0042] as well as

[0043] 3D object detection is performed on the second feature map to output a 3D object detection box.

[0044] In one embodiment of the present invention, the point cloud downsampling module is further configured to perform point cloud downsampling on the original point cloud by the following operations:

[0045] Downsampling the point cloud inside each Pillar of the original point cloud; and

[0046] When the point cloud inside the Pillar exceeds the threshold, a local randomization operation is performed, where:

[0047] The threshold is the number of points N after downsampling in the Pillar multiplied by the expansion coefficient k. The setting of the expansion coefficient k is adjusted according to the randomness of the original point cloud data, and

[0048] The local randomization operation starts from the N+1th point and randomly replaces any point among the N points with a set probability p.

[0049] In one embodiment of the present invention, the BEV encoding module is further configured to perform the BEV encoding by the following operation: compressing the point cloud input features within a Pillar under BEV from the 3D space to the 2DBEV space.

[0050] In one embodiment of the present invention, the BEV encoding module is further configured to perform feature downsampling on the BEV feature map by controlling the step size of the convolutional layer so as to downsample the BEV feature map by a specific multiple, the specific multiple being based on the step size.

[0051] In one embodiment of the present invention, the feature decoding module is further configured to perform the first feature decoding operation by upsampling the downsampled BEV feature map, the first feature map includes the upsampled feature map, and the feature decoding module is further configured to perform the second feature decoding operation by pooling, upsampling and splicing the downsampled BEV feature map, the second feature map includes the spliced ​​feature map, wherein the first feature map has a finer granularity than the second feature map.

[0052] In one embodiment of the present invention, the feature decoding module is further configured to perform BEV multi-label semantic segmentation on the first feature map by the following operations: using a multi-label semantic segmentation loss function to supervise the output of the point cloud semantic segmentation task during the model training phase while keeping the single-label network structure unchanged.

[0053] In one embodiment of the present invention, the system further includes a multi-label post-processing module, which is configured to perform multi-label post-processing on the multi-label BEV semantic mesh result to complete subsequent tasks that require finer-grained point cloud semantic segmentation, wherein the subsequent tasks include curb detection tasks, non-standard obstacle detection tasks, and ground / drivable space extraction tasks.

[0054] In the above embodiment of the present invention, the multi-label post-processing module is further configured to perform the multi-label post-processing by the following operations: performing refinement processing based on the multi-label BEV semantic grid result and point space information, wherein:

[0055] The curb detection task includes combining the grid with the curb and the height jump information of the curb to perform curb linear extraction to output a fine curb.

[0056] The non-standard obstacle detection task includes combining obstacle semantics and the spatial height of the point to output the obstacle feature point semantics, wherein when there are multiple obstacle semantics in the grid, clustering information and adjacent grid semantics are combined to distinguish them, and

[0057] The ground / drivable space extraction task includes ground fitting based on the semantic category of ground elements and the height of the lowest point in the Pillar, and then outputting the drivable space based on the obstacle feature points and ground extraction results.

[0058] In the above embodiment of the present invention, the multi-label post-processing module is further configured to complete the subsequent task by performing the following operations: performing small batch staggered training of heterogeneous label data, including:

[0059] Divide the training data into data with 3D object detection annotations, data with point cloud semantic segmentation annotations, and data with both annotations; and

[0060] The three types of labeled data are polled in small batches with a batch as the smallest unit until all data in the data group with the largest data volume is traversed.

[0061] In another embodiment of the present invention, a computer-readable storage medium is disclosed, which stores instructions for laser multi-task perception for low computing power platforms, including:

[0062] An instruction for receiving an original point cloud and performing point cloud downsampling on the original point cloud;

[0063] Instructions for performing BEV encoding on a point cloud downsampled to generate a BEV feature map;

[0064] Instructions for downsampling the BEV feature map to generate a downsampled BEV feature map;

[0065] Instructions for performing a first feature decoding operation on the downsampled BEV feature map to generate a first feature map;

[0066] Instructions for performing BEV multi-label semantic segmentation on the first feature map to output a multi-label BEV semantic mesh result;

[0067] Instructions for performing a second feature decoding operation on the downsampled BEV feature map to generate a second feature map; and

[0068] Instructions for performing 3D object detection on the second feature map to output a 3D object detection box.

[0069] After studying the description of the specific exemplary embodiments of the present invention below in conjunction with the accompanying drawings, other aspects, features and embodiments of the present invention will be apparent to those of ordinary skill in the art. Although features of the present invention may be discussed below with respect to certain embodiments and accompanying drawings, all embodiments of the present invention may include one or more of the advantageous features discussed herein. In other words, although one or more embodiments may be discussed to have certain advantageous features, one or more of such features may also be used according to the various embodiments of the present invention discussed herein. In a similar manner, although exemplary embodiments may be discussed below as device, system or method embodiments, it should be appreciated that such exemplary embodiments may be implemented in various devices, systems, and methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to understand in detail the manner in which the above-stated features of the present disclosure are used, a more particular description of the content briefly summarized above may be made with reference to various aspects, some of which are illustrated in the accompanying drawings. It should be noted, however, that the accompanying drawings illustrate only certain typical aspects of the present disclosure and are not to be considered limiting of its scope, as the description may admit to other equally effective aspects.

[0071] Figure 1 A schematic block diagram of a system for laser multi-task perception for low-computing-power platforms according to an embodiment of the present disclosure is shown.

[0072] Figure 2 A schematic diagram of a laser multi-task perception model structure according to an embodiment of the present disclosure is shown.

[0073] Figure 3a-3c A schematic block diagram of each network structure block in a laser multi-task perception model structure according to an embodiment of the present disclosure is shown.

[0074] Figure 4 A schematic diagram of a data pipeline for small batch staggered training of heterogeneous labeled data according to an embodiment of the present disclosure is shown.

[0075] Figure 5 A flowchart of a method for laser multi-task perception for low-computing-power platforms according to an embodiment of the present disclosure is shown.

[0076] Figure 6 A schematic diagram of a computer-readable medium configured to store computer-executable instructions according to one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0077] The various embodiments will be described in more detail below with reference to the accompanying drawings that form part of the present invention and illustrate various specific exemplary embodiments. However, the various embodiments can be implemented in many different forms and should not be interpreted as limiting the various embodiments set forth herein; on the contrary, these embodiments are provided to make the present disclosure thorough and complete, and to fully convey the scope of these embodiments to those of ordinary skill in the art. The various embodiments may be implemented in accordance with methods, systems, or devices. Therefore, these embodiments may be implemented in hardware, in full software, or in a combination of software and hardware. Therefore, the following specific embodiments are not limiting.

[0078] The steps in each flow chart may be performed by hardware (e.g., processor, engine, memory, circuit), software (e.g., operating system, application, driver, machine / processor executable instructions), or a combination thereof. As will be appreciated by those skilled in the art, the methods involved in each embodiment may include more or fewer steps than shown.

[0079] In view of the problems in the prior art, a laser multi-task perception method and system for low computing power platforms is proposed. Specifically, the technical solution of the present invention adopts a multi-task system architecture, using one model to simultaneously output 3D target detection boxes and BEV grid multi-label semantic segmentation results, sharing part of the network structure to reduce the amount of model inference calculations and reduce the model's demand for deployment platform computing power.

[0080] In addition, for tasks related to semantic segmentation, the technical solution of the present invention uses a coarse-to-fine processing algorithm to make up for the problem that low-computing power platforms do not have enough resources to perform fine semantic perception of 3D environments, that is, first use a multi-label BEV semantic grid to roughly describe the point cloud categories within the same grid to reduce the amount of model calculation; then use a multi-label post-processing module to further refine the point cloud for different tasks to perform ground / drivable space extraction tasks, curb detection tasks, and non-standard obstacle detection tasks, and in order to perform these tasks, the present invention also uses a small batch staggered training strategy for heterogeneous label data to smooth the training shock caused by the imbalance and missing of multi-task labels. As those skilled in the art can understand, the scenarios to which the low-computing power platform in the present disclosure is applicable may include, but is not limited to, autonomous driving and assisted driving, and may also be used in any other suitable scenarios.

[0081] In addition, when the original point cloud data is received, the technical solution of the present invention first uses an efficient point cloud random sampling algorithm to downsample the received point cloud data to improve the system computing efficiency.

[0082] The various aspects of the present disclosure will be described in more detail and more comprehensively through block diagrams and method flow charts below.

[0083] Figure 1 FIG. 1 is a schematic block diagram of a system 100 for laser multi-task perception for low computing power platforms according to an embodiment of the present disclosure. Figure 2 and Figure 3a-3c Detailed Description Figure 1 System 100 is shown.

[0084] In one embodiment of the present invention, the system 100 may include a point cloud downsampling module 102 , a BEV encoding module 104 , a feature decoding module 106 , and an optional multi-label post-processing module 108 .

[0085] In this embodiment, the point cloud downsampling module 102 can be configured to receive the original point cloud and perform point cloud downsampling on the original point cloud. However, the input point cloud often has poor randomness without random shuffle. When the pillar is divided into large parts and the number of points inside the same pillar is large, the point cloud obtained by sampling will be concentrated in a local area inside the pillar when the points are selected sequentially, and the entire outer contour information of the object cannot be described more completely. Performing global randomization of the point cloud will consume a lot of computing resources when the number of point clouds is large, and when the number of point clouds inside a pillar is not much larger than the number of sampling points, whether to perform point cloud randomization has no effect or little effect on the final point selection. Therefore, the present invention proposes an efficient point cloud random sampling algorithm. The algorithm triggers the local randomization operation only when the point cloud inside the current pillar exceeds the threshold.

[0086] Specifically, the point cloud downsampling module 102 can be further configured to perform point cloud downsampling on the original point cloud through the following operations: downsampling the point cloud inside each Pillar of the original point cloud; and when the point cloud inside the Pillar exceeds the threshold, triggering a local randomization operation. In an exemplary and non-limiting example, the trigger threshold can be the number of points N after downsampling in the current pillar multiplied by the expansion coefficient k (k>=1), where the setting of k can be adjusted according to the randomness of the input point cloud. Therefore, the greater the randomness of the input point cloud, the greater the value of k, the higher the threshold, and the lower the probability of triggering a local randomization operation, ultimately achieving the purpose of saving computing resources and improving the computing efficiency of the system. In an exemplary embodiment of the present invention, the local randomization operation can be to start from N+1 points and randomly replace any one of the N points with a set probability p.

[0087] As can be understood by those skilled in the art, the above-mentioned calculation method of the trigger threshold is exemplary, and any other suitable method may be used to determine the trigger threshold in other embodiments of the present invention, so that when the randomness of the input point cloud is greater, the probability of triggering the local randomization operation is lower. In addition, the above-mentioned local randomization operation is also merely exemplary, and any other suitable local randomization operation may be used in other embodiments of the present invention, without being limited to any specific local randomization operation.

[0088] Figure 2 FIG. 2 is a schematic diagram of a laser multi-task perception model structure 200 according to an embodiment of the present disclosure. Figure 2 In the laser multi-task perception model structure 200 of the present invention, when the original point cloud 202 is received, a local randomization operation 204 of downsampling can be first performed on it, as described above, so that the sampled point cloud can more completely describe the entire outer contour information of the object.

[0089] Back to Figure 1 In one embodiment of the present invention, the BEV encoding module 104 may be configured to: perform BEV encoding on the downsampled point cloud to generate a BEV feature map; and perform feature downsampling on the BEV feature map to generate a downsampled BEV feature map. In the above embodiment of the present invention, the BEV encoding module 104 may be further configured to compress the point cloud input features within a Pillar under BEV from a 3D space to a 2D BEV space to perform a BEV encoding operation.

[0090] like Figure 2 As shown, after the downsampling and local randomization operations are completed, the point cloud after point cloud downsampling can be input feature BEV encoding 206 and feature downsampling can be performed on it (such as Figure 2 As shown in DB 208 in FIG. 1 , DB (DownsamplingBlock) can be interpreted as a feature encoding network structure block) to generate a downsampled BEV feature map. Figure 2 The nx in the formula may indicate that the length and width of the output feature map is 1 / n times of the original input BEV feature map. For example, 4x may indicate that the length and width of the output feature map is 1 / 4 times of the original input BEV feature map.

[0091] About DB 208, Figure 3a FIG. 1 shows a schematic block diagram of the network structure block. In an exemplary embodiment of the present invention, the DB structure block has one input and two outputs, namely, the BEV-encoded feature map is taken as input, and the output is downsampled (such as pooling, such as Figure 3a Features shown) Figure 1 And the un-downsampled features Figure 2 .

[0092] Figure 2 The local randomization 204, input feature BEV encoding 208 and DB 208 can be included in the multi-task shared network structure (also called the multi-task shared feature encoding network layer) in the laser multi-task perception model structure 200 for extracting features of different dimensions, thereby facilitating the subsequent use of different feature decoding network structures to complete different decoding tasks.

[0093] Back to Figure 1 In one embodiment of the present invention, the BEV encoding module 104 may be further configured to perform feature downsampling on the generated BEV feature map by the following operations: controlling the stride of the convolution layer so as to downsample the BEV feature map by a specific multiple, the specific multiple being based on the stride. As an example and not a limitation, the stride of the convolution layer may be set to 2, and a 2-fold downsampling may be performed so that the length and width of the feature map are halved. As can be appreciated by those skilled in the art, the above-mentioned feature downsampling method is merely exemplary and not restrictive, and any other suitable feature downsampling method may be used in other embodiments of the present invention to perform feature downsampling on the generated BEV feature map.

[0094] In one embodiment of the present invention, the feature decoding module 106 may be configured to perform a first feature decoding operation on the downsampled BEV feature map to generate a first feature map, and perform BEV multi-label semantic segmentation on the first feature map to output a multi-label BEV semantic mesh result. In an exemplary embodiment, the feature decoding module 106 may be further configured to generate an upsampled feature map by upsampling the downsampled BEV feature map. Figure 2 As shown, as an example but not limitation, the downsampled BEV feature map can be upsampled by UB (Upsampling Block) 210 as a feature decoding network structure block to generate an upsampled feature map. Subsequently, the upsampled feature map can be subjected to BEV multi-label semantic segmentation by, for example, a BEV multi-label semantic segmentation head to output a multi-label BEV semantic grid result.

[0095] About UB 210, Figure 3b . In an exemplary embodiment of the present invention, the UB block has two inputs and one output, and performs upsampling (such as 2x deconvolution, such as Figure 3b As shown) and concatenate the two input feature maps and then generate an output feature map through any suitable neural network block.

[0096] The label of a BEV grid represents the description of the semantic information of all points in the pillar. Compared with the semantic segmentation results of the output 3Dvoxel (voxel) granularity, the coarse-grained BEV pillar can effectively reduce the amount of calculation. However, the coarser the grid granularity, the more point cloud categories are contained in a single pillar, and the conventional single label cannot fully describe the semantic state inside the pillar. Therefore, the present invention uses multiple labels to describe the semantic state in the BEV pillar, and better expresses the semantic information of the point cloud without increasing the computational complexity of the reasoning model.

[0097] In another embodiment of the present invention, the feature decoding module 106 can be further configured to perform BEV multi-label semantic segmentation on the first feature map by adopting a multi-label semantic segmentation loss function to supervise the output of the point cloud semantic segmentation task during the model training phase while keeping the single-label network structure unchanged.

[0098] Back to Figure 1 , the feature decoding module 106 may also be configured to perform a second feature decoding operation on the downsampled BEV feature map to generate a second feature map, and perform 3D object detection on the second feature map to output a 3D object detection frame. In an exemplary embodiment, the feature decoding module 106 may be further configured to generate a spliced ​​feature map by pooling, upsampling, and splicing the downsampled BEV feature map, whereby the second feature map (i.e., the spliced ​​feature map) has a coarser granularity than the first feature map (i.e., the upsampled feature map).

[0099] like Figure 2 As shown, by way of example and not limitation, the downsampled BEV feature map may be pooled, upsampled and concatenated to generate a concatenated feature map by pooling, upsampling (UP) and concatenation as three different feature decoding network structural blocks 212. Subsequently, 3D object detection may be performed on the concatenated feature map by, for example, a 3D object detection head to output a 3D object detection frame.

[0100] About UP structure block, Figure 3c In an exemplary embodiment of the present invention, the UP module has one input and one output, that is, the feature map downsampled by the DB structure block is taken as input and upsampled (such as 2x deconvolution, such as Figure 3c As shown) to output an upsampled feature map.

[0101] As can be understood by those skilled in the art, Figure 3a-3c The three network structure blocks shown are merely exemplary and non-limiting. In other embodiments of the present invention, any other suitable network structure blocks may be used to implement the same or similar functions.

[0102] In summary, the 3D object detection task can be performed on a coarser-grained feature map, while the BEV multi-label semantic segmentation task can be performed on a relatively fine-grained feature map to generate a multi-label BEV semantic mesh result. In summary, the present invention adopts a multi-task system architecture, using one model to simultaneously output 3D object detection boxes and BEV mesh multi-label semantic segmentation results, sharing part of the network structure to reduce the amount of model inference calculations and reduce the model's demand for computing power on the deployment platform.

[0103] As those skilled in the art will appreciate, the above-mentioned first and second feature decoding operations and the generated first and second feature maps are merely exemplary and non-restrictive. In other embodiments of the present invention, any other suitable feature decoding operations may be performed on the downsampled BEV feature map or other different feature decoding network structures may be used to generate any other suitable feature maps suitable for other perception tasks, rather than being limited to the first and second feature maps. Figure 1 and Figure 2 The decoding operation is shown.

[0104] Back to Figure 1 In another embodiment of the present invention, the system 100 may also optionally include a multi-label post-processing module 108, which may be configured to perform multi-label post-processing on the generated multi-label BEV semantic mesh result to complete subsequent tasks that require finer-grained point cloud semantic segmentation. In this embodiment, the subsequent tasks may include a curb detection task, a non-standard obstacle detection task, and a ground / freespace extraction task. In other embodiments of the present invention, the subsequent tasks may also include any other applicable tasks.

[0105] In another embodiment of the present invention, the multi-label post-processing module 108 can be further configured to perform the multi-label post-processing through the following operations: refinement processing based on the multi-label BEV semantic grid results and point spatial information, wherein: the curb detection task includes combining the grid where the curb exists and the height jump information of the curb to perform curb linear extraction to output a fine curb, the non-standard obstacle detection task includes combining the obstacle semantics and the spatial height of the point above the ground to output the obstacle feature point semantics, wherein when there are multiple obstacle semantics in the grid, they are distinguished by combining the clustering information and the semantics of adjacent grids, and the ground / drivable space extraction task includes combining the ground element semantic category and the lowest point height in the Pillar to perform ground fitting, and then combining the obstacle feature points and the ground extraction results to output the drivable space.

[0106] Therefore, for semantic segmentation related tasks, the present invention uses a coarse-to-fine processing algorithm to make up for the problem that low-computing platforms do not have enough resources to perform fine semantic perception of the 3D environment. First, a multi-label BEV semantic grid is used to roughly describe the point cloud categories within the same grid to reduce the amount of model calculation. Then, the multi-label post-processing module 108 further refines the point cloud for different tasks to complete the ground / movable space extraction task, the curb detection task, and the non-standard obstacle detection task.

[0107] In order to cope with the following application scenarios: the labeled data of different tasks (such as the above three tasks) have diversity problems (some data only have point cloud semantic segmentation annotations, some data only have 3D object detection annotations, and some data contain both annotations), and the amount of labeled data for different tasks is extremely unbalanced. In order to make full use of all data and mine the value of all data, and smooth the training shock caused by the imbalance and missing of multi-task labels, the present invention uses a small batch staggered training strategy for heterogeneous labeled data.

[0108] Specifically, in another embodiment of the present invention, the multi-label post-processing module 108 can be further configured to complete subsequent tasks by performing small batch interleaved training of heterogeneous labeled data, including: dividing the training data into data with 3D target detection annotations, data with point cloud semantic segmentation annotations, and data with both annotations; and polling these three types of labeled data in small batches with a batch as the minimum unit until the data group with the largest data volume completes all data traversal.

[0109] Figure 4 A schematic diagram of a data pipeline for small batch staggered training of heterogeneous labeled data according to an embodiment of the present disclosure is shown. Figure 4 The data pipeline in one epoch during the entire training process is shown: First, all data is divided into three groups, namely data with 3D object detection annotations, data with point cloud semantic segmentation annotations, and data with both annotations; then these three types of annotated data are polled in small batches with a batch as the minimum unit, where N1, N2, and N3 are the polling coefficients. Finally, when the data group with the largest amount of data completes all data traversal, it is considered that the training of one epoch is completed. During the polling, the data group with a small amount of data is repeatedly traversed.

[0110] Figure 5 A flowchart of a method 500 for laser multi-task perception for low-computing-power platforms according to an embodiment of the present disclosure is shown.

[0111] like Figure 5As shown, method 500 starts at step 502, receiving an original point cloud and performing point cloud downsampling on the original point cloud. In one embodiment of the present invention, performing point cloud downsampling on the original point cloud may further include: downsampling the point cloud inside each Pillar of the original point cloud; and when the point cloud inside the Pillar exceeds a threshold, performing a local randomization operation, wherein: the threshold is the number of points N after downsampling in the Pillar multiplied by a dilation coefficient k, the setting of the dilation coefficient k is adjusted according to the randomness of the original point cloud data, and the local randomization operation starts from the N+1th point and randomly replaces any point among the N points with a set probability p.

[0112] Next, the method 500 proceeds to step 504, performing BEV encoding on the downsampled point cloud to generate a BEV feature map. In one embodiment of the present invention, this step may include compressing the point cloud input features within a Pillar under BEV from a 3D space to a 2D BEV space.

[0113] Subsequently, the method 500 proceeds to step 506, where the BEV feature map is feature downsampled to generate a downsampled BEV feature map. In one embodiment of the present invention, this step may further include controlling the step size of the convolutional layer so as to downsample the BEV feature map by a specific multiple, the specific multiple being based on the step size.

[0114] Then, method 500 proceeds to step 508, performs a first feature decoding operation on the downsampled BEV feature map to generate a first feature map, and proceeds to step 510, performs BEV multi-label semantic segmentation on the first feature map to output a multi-label BEV semantic mesh result. In one embodiment of the present invention, the first feature decoding operation includes upsampling the downsampled BEV feature map, whereby the first feature map includes the upsampled feature map. In one embodiment of the present invention, step 510 may further include using a multi-label semantic segmentation loss function to supervise the output of the point cloud semantic segmentation task during the model training phase while maintaining the same structure relative to the single-label network.

[0115] Next, the method 500 proceeds to step 512, performs a second feature decoding operation on the downsampled BEV feature map to generate a second feature map, and finally proceeds to step 514, performs 3D object detection on the second feature map to output a 3D object detection frame. In one embodiment of the present invention, the second feature decoding operation includes pooling, upsampling, and concatenating the downsampled BEV feature map, whereby the second feature map includes a concatenated feature map, wherein the first feature map has a finer granularity than the second feature map.

[0116] After step 514 , method 500 ends.

[0117] In another embodiment of the present invention, the method 500 may optionally further include multi-label post-processing of the multi-label BEV semantic grid result to complete subsequent tasks that require finer-grained point cloud semantic segmentation, wherein the subsequent tasks may include curb detection tasks, non-standard obstacle detection tasks, and ground / drivable space extraction tasks. In the above embodiment of the present invention, the multi-label post-processing may include refinement processing based on the multi-label BEV semantic grid result and point space information, wherein: the curb detection task may include combining the grid with the curb and the height jump information of the curb to perform curb linear extraction to output a fine curb, the non-standard obstacle detection task may include combining the obstacle semantics and the spatial height of the point to output the obstacle feature point semantics, wherein when there are multiple obstacle semantics in the grid, the clustering information and the adjacent grid semantics are combined for distinction, and the ground / drivable space extraction task may include combining the ground element semantic category and the lowest point height in the Pillar to perform ground fitting, and then combining the obstacle feature points and the ground extraction results to output the drivable space. In the above-mentioned embodiment of the present invention, completing the subsequent task further includes performing small batch interleaved training of heterogeneous labeled data, including: dividing the training data into data with 3D target detection annotations, data with point cloud semantic segmentation annotations, and data with both annotations; and polling the three types of annotated data in small batches with a batch as the minimum unit until the data group with the largest data volume completes all data traversal.

[0118] Figure 6 A schematic diagram of a computer-readable storage medium 606 configured to store computer-executable instructions 604 is shown according to one embodiment 600 of the present disclosure.

[0119] like Figure 6 As shown, in this embodiment 600, an exemplary computer-readable storage medium 606 is provided. Computer-executable instructions 604 (hereinafter also referred to as instructions) are stored on the computer-readable storage medium 606. In this embodiment 600, the instructions 604 may be configured to implement a method (as shown in the dashed box 602) when executed by a processor, such as Figure 5 In another embodiment, the instructions 604 may also be configured to implement a system such as Figure 1 The system 100 shown. As can be appreciated by those skilled in the art, Figure 6The step boxes in the dashed box 602 in are merely exemplary and non-restrictive. A person of ordinary skill in the art can design many such computer-readable media that can be configured to perform the desired method or implement the desired system. The term "computer-readable storage medium" used herein includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-executable instructions or other data. Memory and storage are examples of computer-readable storage media. Computer-readable storage media can be non-transient and include, but are not limited to: RAM, ROM, EEPROM, flash memory or other storage technology, CD-ROM, digital versatile disk (DVD) or other optical storage, cassette, tape, disk storage or other magnetic storage device, or any other medium that can be used to store the desired information and can be accessed by a computing device.

[0120] Although not required, the various embodiments are described in the general context of "computer-executable instructions" executed by one or more processors. Computer-executable instructions can be distributed via computer-readable storage media. Computer-executable instructions can be implemented as program modules, such as functions, objects, application programming interfaces (APIs), data structures, etc., that perform a particular method or implement a particular system. In general, the functionality of the computer-executable instructions can be combined or distributed in various environments as needed.

[0121] The term "computer-readable storage medium" may also include communication media. Communication media typically embody computer-executable instructions or other data in a "modulated data signal" such as a carrier wave or other transmission mechanism, and includes any information transmission medium. The term "modulated data signal" may include a signal that sets or changes one or more of its characteristics in a manner that encodes the information in the signal. Those skilled in the art will recognize that computer-readable storage media used to store computer-executable instructions can be distributed over a network. For example, a computer-readable storage medium accessible via a network may store computer-executable instructions to implement one or more embodiments provided by the present disclosure. A computing device may access a computer-readable storage medium over a network and download part or all of the computer-executable instructions for execution. Alternatively, some instructions may be executed on one computing device and other instructions on another computing device.

[0122] In summary, the technical solution of the present invention adopts a multi-task system architecture, uses one model to simultaneously output 3D target detection boxes and BEV grid multi-label semantic segmentation results, shares part of the network structure to reduce the amount of model inference calculations, and reduces the model's demand for computing power on the deployment platform.

[0123] Embodiments of the present invention have been described above with reference to block diagrams and / or the operating instructions of methods, systems, and computer program products according to embodiments of the present invention. Each function / action noted in the frame may occur in an order different from that shown in any flow chart. For example, depending on the function / action involved, two frames shown in succession may actually be performed substantially simultaneously, or these frames may sometimes be performed in an opposite order.

[0124] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for laser multi-task perception for a low computing power platform, the method comprising: Receiving an original point cloud and performing point cloud downsampling on the original point cloud; Performing BEV encoding on the downsampled point cloud to generate a BEV feature map; Downsampling the BEV feature map to generate a downsampled BEV feature map; performing a first feature decoding operation on the downsampled BEV feature map to generate a first feature map; Performing BEV multi-label semantic segmentation on the first feature map to output a multi-label BEV semantic grid result; performing a second feature decoding operation on the downsampled BEV feature map to generate a second feature map; as well as Perform 3D object detection on the second feature map to output a 3D object detection box.

2. The method of claim 1, wherein downsampling the original point cloud further comprises: Downsampling the point cloud inside each Pillar of the original point cloud; as well as When the point cloud inside the Pillar exceeds the threshold, a local randomization operation is performed, where: The threshold is the number of points N after downsampling in the Pillar multiplied by the expansion coefficient k, and the setting of the expansion coefficient k is adjusted according to the randomness of the original point cloud data, and The local randomization operation starts from the N+1th point and randomly replaces any point among the N points with a set probability p.

3. The method of claim 1, wherein: The first feature decoding operation includes upsampling the downsampled BEV feature map, the first feature map includes the upsampled feature map, and The second feature decoding operation includes pooling, upsampling and concatenating the downsampled BEV feature map, the second feature map including a concatenated feature map, wherein the first feature map has a finer granularity than the second feature map.

4. The method as claimed in claim 1, wherein the method further comprises performing multi-label post-processing on the multi-label BEV semantic mesh result to complete subsequent tasks that require finer-grained point cloud semantic segmentation, wherein the subsequent tasks include curb detection tasks, non-standard obstacle detection tasks, and ground / drivable space extraction tasks.

5. The method of claim 4, wherein the multi-label post-processing comprises performing refinement processing based on the multi-label BEV semantic grid result and point space information, wherein: The curb detection task includes combining the grid where the curb exists and the height jump information of the curb to perform curb linear extraction to output a fine curb. The non-standard obstacle detection task includes combining obstacle semantics and the spatial height of the point to output obstacle feature point semantics, wherein when there are multiple obstacle semantics in the grid, clustering information and adjacent grid semantics are combined to distinguish them, and The ground / drivable space extraction task includes fitting the ground by combining the semantic category of ground elements and the height of the lowest point in the Pillar, and then outputting the drivable space by combining the obstacle feature points and the ground extraction result.

6. The method of claim 4, wherein completing the subsequent task further comprises performing small batch staggered training of heterogeneous label data, comprising: The training data is divided into data with 3D object detection annotations, data with point cloud semantic segmentation annotations, and data with both annotations; as well as The three types of labeled data are polled in small batches with a batch as the smallest unit until all data in the data group with the largest data volume is traversed.

7. A system for laser multi-task perception for low computing power platforms, the system comprising: A point cloud downsampling module, configured to receive an original point cloud and perform point cloud downsampling on the original point cloud; The BEV encoding module is configured to: Performing BEV encoding on the downsampled point cloud to generate a BEV feature map; and Downsampling the BEV feature map to generate a downsampled BEV feature map; a feature decoding module, which is configured to: performing a first feature decoding operation on the downsampled BEV feature map to generate a first feature map; Performing BEV multi-label semantic segmentation on the first feature map to output a multi-label BEV semantic grid result; performing a second feature decoding operation on the downsampled BEV feature map to generate a second feature map; as well as Perform 3D object detection on the second feature map to output a 3D object detection box.

8. The system of claim 6, wherein the point cloud downsampling module is further configured to perform point cloud downsampling on the original point cloud by: Downsampling the point cloud inside each Pillar of the original point cloud; and When the point cloud inside the Pillar exceeds the threshold, a local randomization operation is performed, where: The threshold is the number of points N after downsampling in the Pillar multiplied by the expansion coefficient k, and the setting of the expansion coefficient k is adjusted according to the randomness of the original point cloud data, and The local randomization operation starts from the N+1th point and randomly replaces any point among the N points with a set probability p.

9. The system of claim 6, wherein: The feature decoding module is further configured to perform the first feature decoding operation by upsampling the downsampled BEV feature map, the first feature map comprising the upsampled feature map, and The feature decoding module is further configured to perform the second feature decoding operation by pooling, upsampling and concatenating the downsampled BEV feature map, the second feature map comprising a concatenated feature map, wherein the first feature map has a finer granularity than the second feature map.

10. The system as claimed in claim 6, wherein the system further includes a multi-label post-processing module, which is configured to perform multi-label post-processing on the multi-label BEV semantic mesh result to complete subsequent tasks that require finer-grained point cloud semantic segmentation, wherein the subsequent tasks include curb detection tasks, non-standard obstacle detection tasks, and ground / drivable space extraction tasks.

11. The system of claim 10, wherein the multi-label post-processing module is further configured to perform the multi-label post-processing by performing a refinement process based on the multi-label BEV semantic grid result and point space information, wherein: The curb detection task includes combining the grid where the curb exists and the height jump information of the curb to perform curb linear extraction to output a fine curb. The non-standard obstacle detection task includes combining obstacle semantics and the spatial height of the point to output obstacle feature point semantics, wherein when there are multiple obstacle semantics in the grid, clustering information and adjacent grid semantics are combined to distinguish them, and The ground / drivable space extraction task includes fitting the ground by combining the semantic category of ground elements and the height of the lowest point in the Pillar, and then outputting the drivable space by combining the obstacle feature points and the ground extraction result.

12. The system of claim 10, wherein the multi-label post-processing module is further configured to complete the subsequent task by performing small batch interleaved training of heterogeneous label data, comprising: The training data is divided into data with 3D object detection annotations, data with point cloud semantic segmentation annotations, and data with both annotations; as well as The three types of labeled data are polled in small batches with a batch as the smallest unit until all data in the data group with the largest data volume is traversed.

13. A computer-readable storage medium storing thereon instructions for laser multi-task perception for low computing power platforms, wherein the instructions, when executed by a processor, implement the method of any one of claims 1 to 6.

Images