Method and apparatus for detecting 3D point cloud target

Abstract

The application discloses a 3D point cloud target detection method and device. The method comprises the following steps: obtaining three-dimensional point cloud data of a driving road during driving of an automatic driving vehicle, wherein the automatic driving vehicle is provided with a low-power embedded platform, and the three-dimensional point cloud data of the driving road is collected through a laser radar sensor installed on the automatic driving vehicle; inputting the three-dimensional point cloud data of the driving road into an optimized model of a 3D point cloud target detection model, and identifying at least one target information located in a region range of the driving road; wherein the optimized model of the 3D point cloud target detection model is a model running on the low-power embedded platform. The application solves the technical problem of small use range of the 3D target detection model.

Description

Technical Field

[0001] This invention relates to the field of vehicles, and more specifically, to a method and apparatus for detecting 3D point cloud targets. Background Technology

[0002] Currently, in order to meet the needs of environmental perception, autonomous vehicles are often equipped with LiDAR sensors to acquire real-time 3D point cloud data of the road. This device can be used to input the point cloud data into a 3D point cloud target detection algorithm. The deep learning-based 3D point cloud target detection algorithm realizes end-to-end perception function, thereby obtaining various target information of the surrounding environment and providing accurate information for the environmental perception of autonomous vehicles.

[0003] In autonomous driving solutions based on LiDAR sensors, general-purpose chips such as graphics processing units (GPUs) still dominate. However, while GPUs are versatile, fast, and efficient, they also consume a lot of power, making them unsuitable for autonomous vehicles. This has led to the development of AI processors specifically designed for autonomous driving. Although these AI processors offer high computing power, high performance, and low power consumption, their complex design and high demands on model portability mean that even with the emergence of many high-precision 3D object detection models, limitations in computing power, portability, and accuracy often prevent their widespread adoption in autonomous vehicles, resulting in the technical challenge of limited applicability of 3D object detection models.

[0004] The existing technologies mentioned above have limited applicability of 3D target detection models, and no effective solution has yet been proposed. Summary of the Invention

[0005] This invention provides a method and apparatus for detecting 3D point cloud targets, which at least solves the technical problem of the limited application range of 3D target detection models.

[0006] According to one aspect of the present invention, a method for detecting a 3D point cloud target is provided, comprising: obtaining an original model of a 3D point cloud target detection model; obtaining a backbone network model by pruning the original model; preprocessing the backbone network model by combining lightweight point cloud features to generate a target model; and generating an overall error of the target model based on the detection error of the original model and the detection error of the target model.

[0007] Optionally, when controlling the braking of the target vehicle based on the target control data, the method further includes: using a pruning algorithm to prune the heavyweight backbone network part in the original model to obtain a backbone network model, wherein the backbone network model is the lightweight backbone network in the target model.

[0008] Optionally, the original model is trained iteratively for multiple rounds based on a preset loss function. When the model accuracy of the original model reaches the target accuracy value after multiple rounds of iterative training, an optimized original model is generated. The loss function is a function based on the stochastic gradient descent algorithm.

[0009] Optionally, before inputting the 3D point cloud data of the driving road into the optimized model of the 3D point cloud target detection model, the method further includes: after performing multiple rounds of iterative training on the original model, if the overall error of the target model decreases to the target value, verifying the model accuracy of the original model; if the model accuracy reaches the target accuracy value, determining that the accuracy of the target model also reaches the target accuracy value; performing fixed-point compression on the target model that has reached the target accuracy value to generate the optimized model of the 3D point cloud target detection model.

[0010] Optionally, based on the detection error of the original model and the detection error of the target model, the overall error of the target model is generated, including: obtaining the detection error L of the original model. teacher , where L teacher =L reg +λL cls L reg To detect the error between the bounding box regression value and the annotation information, L cls The error between the bounding box classification value and the annotation information is represented by λ, which denotes the weight of the detection branch of the original model; the detection error L of the target model is obtained. student , where L student =L reg +λ1L cls L reg To detect the error between the bounding box regression value and the annotation information, L cls To detect errors in bounding box classification values ​​and annotation information, λ1 represents the weight of the target model branch; the classification feature maps of the original model and the target model are passed through activation layers respectively, and the root mean square error L is calculated. hm The absolute error L between the original model and the target model is calculated by using the detection head to measure the dimensions and center points of the two models. wlh and L xyz Based on the root mean square error, the absolute value of the error between the original model and the target model L is calculated. wlh and L xyz Generate the distillation error L between the two models. kd =L hm +λ2L xyz +λ3L wlh During the multiple rounds of iterative training of the original model, based on the distillation error, the target model is simultaneously trained in multiple rounds to obtain the overall error of the target model. When the overall error of the target model decreases to the target value, if the model accuracy reaches the target accuracy value, it is determined that the accuracy of the target model has also reached the target accuracy value.

[0011] According to another aspect of the present invention, a 3D point cloud target detection device is also provided, comprising: a first acquisition module, configured to acquire three-dimensional point cloud data of a driving road during the driving of an autonomous vehicle, wherein the autonomous vehicle is equipped with a low-power embedded platform and acquires the three-dimensional point cloud data of the driving road through an installed lidar sensor; and an identification module, configured to input the three-dimensional point cloud data of the driving road into an optimized model of a 3D point cloud target detection model to identify at least one target information located within the area of ​​the driving road; wherein the optimized model of the 3D point cloud target detection model is a model running on a low-power embedded platform.

[0012] Optionally, the device further includes: a second acquisition module for acquiring the original model of the 3D point cloud target detection model; a pruning module for obtaining a backbone network model by pruning the original model; a preprocessing module for preprocessing the backbone network model by combining lightweight point cloud features to generate a target model; and a generation module for generating the overall error of the target model based on the detection error of the original model and the detection error of the target model.

[0013] Optionally, the pruning module includes a sub-pruning module, which is used to prune the heavyweight backbone network part in the original model using a pruning algorithm to obtain a backbone network model, wherein the backbone network model is the lightweight backbone network in the target model.

[0014] Optionally, the device further includes: a training module for performing multiple rounds of iterative training on the original model based on a preset loss function, and generating an optimized original model when the model accuracy of the original model reaches the target accuracy value after multiple rounds of iterative training, wherein the loss function is a function based on the stochastic gradient descent algorithm.

[0015] According to another aspect of the present invention, a computer-readable storage medium is also provided. The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the 3D point cloud target detection method of the present invention.

[0016] According to another aspect of the present invention, a processor is also provided. The processor is used to run a program, wherein the program executes the 3D point cloud target detection method of the present invention during runtime.

[0017] In this embodiment of the invention, during the operation of an autonomous vehicle, three-dimensional point cloud data of the driving road is acquired. The autonomous vehicle is equipped with a low-power embedded platform, and the three-dimensional point cloud data of the driving road is collected by an installed LiDAR sensor. The three-dimensional point cloud data of the driving road is input into an optimized model of a 3D point cloud target detection model to identify at least one target located within the area of ​​the driving road. The optimized model of the 3D point cloud target detection model runs on the low-power embedded platform. In other words, this application, based on a low-power embedded platform, acquires three-dimensional point cloud data of the driving road through an installed LiDAR sensor during the operation of an autonomous vehicle. By inputting this data into an optimized model of a 3D point cloud target detection model on the low-power embedded platform, at least one target located within the area of ​​the driving road is identified. This achieves the technical effect of expanding the application range of the 3D target detection model and solves the technical problem of the limited application range of the 3D target detection model. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0019] Figure 1 This is a flowchart of a 3D point cloud target detection method according to an embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram of a 3D point cloud target detection device according to an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] Example 1

[0024] According to an embodiment of the present invention, an embodiment of a method for detecting 3D point cloud targets is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0025] Figure 1 This is a flowchart of a 3D point cloud target detection method according to an embodiment of the present invention, such as... Figure 1 The flowchart shown is for a 3D point cloud target detection method, which includes the following steps:

[0026] Step S102: During the autonomous driving process, acquire three-dimensional point cloud data of the driving road.

[0027] In the technical solution provided in step S102 of the present invention, an autonomous vehicle is equipped with a low-power embedded platform, which collects three-dimensional point cloud data of the driving road through the installed lidar sensor.

[0028] In this embodiment, the embedded platform is a dedicated computer system whose hardware and software modules can be flexibly tailored according to user needs (such as functionality, reliability, cost, size, power consumption, environment, etc.). The lidar sensor measures the propagation distance between the sensor transmitter and the target object, analyzes the magnitude of the reflected energy from the object's surface, and information such as the amplitude, frequency, and phase of the reflected spectrum, thereby presenting three-dimensional point cloud data collected during the autonomous vehicle's operation. The lidar sensor can be a binocular camera, a 3D scanner, etc.

[0029] Optionally, the 3D point cloud data of the driving road is a set of data of points in a certain coordinate system, including 3D coordinates X, Y, Z, color, classification value, intensity value, time, etc., obtained through a LiDAR sensor. For example, 3D point cloud data can be created by scanning images captured by a binocular camera and the camera's intrinsic parameters.

[0030] Optionally, the collected three-dimensional point cloud data of the driving road can be information such as the status of the autonomous vehicle, traffic flow information, road conditions, and traffic signs.

[0031] Step S104: Input the three-dimensional point cloud data of the driving road into the optimization model of the 3D point cloud target detection model to identify at least one target information located within the area where the driving road is located.

[0032] In the technical solution provided by step S104 of the present invention, the three-dimensional point cloud data of the driving road is input into the optimization model of the 3D point cloud target detection model to identify at least one target information located in the area where the driving road is located. The optimization model of the 3D point cloud target detection model is a model running on a low-power embedded platform.

[0033] Optionally, the optimized model of the 3D point cloud target detection model can be a model built according to the system requirements and running on a low-power embedded platform. For example, the input is 3D point cloud data of the driving road, and the optimized model is obtained through data processing.

[0034] Optionally, the target information can be the status of the autonomous vehicle, traffic flow information, road conditions, traffic signs, etc., or information such as pedestrians, vehicles, and buildings encountered by the autonomous vehicle during its operation. No specific restrictions are imposed here.

[0035] In steps S102 to S104 of this application, during the operation of an autonomous vehicle, three-dimensional point cloud data of the driving road is acquired. The autonomous vehicle is equipped with a low-power embedded platform, and the three-dimensional point cloud data of the driving road is collected through an installed LiDAR sensor. The three-dimensional point cloud data of the driving road is input into an optimized model of a 3D point cloud target detection model to identify at least one target located within the area of ​​the driving road. The optimized model of the 3D point cloud target detection model runs on the low-power embedded platform. In other words, this application, based on a low-power embedded platform, acquires three-dimensional point cloud data of the driving road through an installed LiDAR sensor during the operation of an autonomous vehicle. By inputting this data into an optimized model of a 3D point cloud target detection model on the low-power embedded platform, at least one target located within the area of ​​the driving road is identified. This achieves the technical effect of expanding the application range of the 3D target detection model and solves the technical problem of the limited application range of the 3D target detection model.

[0036] The method described in this embodiment will be further described below.

[0037] As an optional embodiment, the method further includes: obtaining the original model of the 3D point cloud target detection model; obtaining the backbone network model by pruning the original model; preprocessing the backbone network model with lightweight point cloud features to generate the target model; and generating the overall error of the target model based on the detection error of the original model and the detection error of the target model.

[0038] In this embodiment, the original model, also known as the teacher model, is used to guide the training of other models. This model can include three-dimensional coordinates X, Y, and Z, as well as intensity values ​​and time, and is constructed by combining 3D sparse convolutional coding, a heavyweight backbone network, and multiple detection heads. The 3D sparse convolutional coding is used to reconstruct accurate information encoding from sparse and irregular data. The detection head consists of a classification branch and a regression branch. The classification branch represents the type of detection information, such as people, vehicles, and houses; the regression branch represents the features of the detection information, such as length, width, height, depth, and orientation angle.

[0039] Optionally, the backbone network model can be obtained by pruning the original model. Pruning simplifies the model, thereby speeding up its execution. This can be achieved by pruning algorithms that remove low-weight connections, thus simplifying the model. The backbone network model can be a model that retains key points. For example, by retaining the key points of the original input and pruning unimportant connections, a network model with only key points is obtained, reducing the model size and improving algorithm efficiency and runtime.

[0040] Optionally, this embodiment combines lightweight point cloud features to preprocess the backbone network model to generate the target model. The preprocessing can involve processing the point cloud grid from a bird's-eye view, with the cropped backbone network model and the detection head together forming the target model. This target model is also known as a 3D point cloud target detection model.

[0041] As an optional embodiment, when controlling the braking of the target vehicle based on target control data, the method further includes: using a pruning algorithm to prune the heavyweight backbone network part in the original model to obtain a backbone network model, wherein the backbone network model is the lightweight backbone network in the target model.

[0042] Alternatively, the pruning algorithm can process the graph to display appropriate areas, for example, testing points with rendering capabilities and removing points if they are outside the boundaries.

[0043] As an optional implementation method, the original model is trained iteratively for multiple rounds based on a preset loss function. When the model accuracy of the original model reaches the target accuracy value after multiple rounds of iterative training, an optimized original model is generated. The loss function is a function based on the stochastic gradient descent algorithm.

[0044] In this embodiment, a loss function can be set for the original model. The loss function is used to minimize the loss function and evaluate the model. The original model is trained using stochastic gradient descent. Based on the preset loss function, the original model is trained iteratively for multiple rounds. When the accuracy of the trained original model reaches the target accuracy value, an optimized original model is generated. The optimized original model includes a higher-accuracy 3D point cloud object detection student model and a higher-accuracy 3D point cloud object detection teacher model.

[0045] Optionally, the target accuracy can be a specific accuracy value of the input, or it can be set to indicate that the model accuracy of the original model has reached the target accuracy value when the model accuracy of the original model no longer increases after multiple rounds of iterative training.

[0046] For example, in machine learning algorithms, a loss function is constructed for the original model, and then the loss function is optimized using a gradient descent-based optimization algorithm to minimize the value of the loss function. When the model accuracy of the original model no longer increases during training, it means that the model accuracy of the original model after training has reached the target accuracy value, thus generating the optimized original model.

[0047] As an optional implementation, before inputting the 3D point cloud data of the driving road into the optimized model of the 3D point cloud target detection model, the method further includes: after performing multiple rounds of iterative training on the original model, if the overall error of the target model decreases to the target value, verifying the model accuracy of the original model; if the model accuracy reaches the target accuracy value, determining that the accuracy of the target model also reaches the target accuracy value; performing fixed-point compression on the target model that has reached the target accuracy value to generate the optimized model of the 3D point cloud target detection model.

[0048] In this embodiment, after multiple rounds of iterative training on the original model, if the overall error of the target model decreases to the target value, the model accuracy of the original model is verified. If the model accuracy reaches the target accuracy value, the target model that has reached the target accuracy value is subjected to fixed-point compression to generate an optimized model of the 3D point cloud target detection model.

[0049] In this embodiment, the overall error of the target model can also be called the cross-entropy loss function. The target value for the reduction of the overall error can be a specific value that is input in advance, or it can be set to the value when the overall error of the original model no longer decreases after multiple rounds of iterative training.

[0050] Optionally, the optimized model of the 3D point cloud target detection model can be a 3D point cloud target detection model that can be output by performing fixed-point compression on the target model using a quantization perception training method, and finally outputting a 3D point cloud target detection model that can run on a low-power embedded device.

[0051] As an optional implementation method, the overall error of the target model is generated based on the detection error of the original model and the detection error of the target model, including: obtaining the detection error L of the original model. teacher , where L reacher =L reg +λL cls L reg To detect the error between the bounding box regression value and the annotation information, L cls The error between the bounding box classification value and the annotation information is represented by λ, which denotes the weight of the detection branch of the original model; the detection error L of the target model is obtained. student , where L student =L reg +λ1L cls L reg To detect the error between the bounding box regression value and the annotation information, L cls To detect errors in bounding box classification values ​​and annotation information, λ1 represents the weight of the target model branch; the classification feature maps of the original model and the target model are passed through activation layers respectively, and the root mean square error L is calculated. hm The absolute error L between the original model and the target model is calculated by using the detection head to measure the dimensions and center points of the two models. wlh and Lxyz Based on the root mean square error, the absolute value of the error between the original model and the target model L is calculated. wlh and L xyz Generate the distillation error L between the two models. kd =L hm +λ2L xyz +λ3L wlh During the multiple rounds of iterative training of the original model, based on the distillation error, the target model is simultaneously subjected to multiple rounds of iterative training to obtain the overall error of the target model. When the overall error of the target model decreases to the target value, if the model accuracy reaches the target accuracy value, it is determined that the accuracy of the target model has also reached the target accuracy value.

[0052] In this embodiment, the original model detection error L is obtained. teacher L teacher =L reg +λL cls Lreg is the error between the regression value and the annotation information of the detection box, Lcls is the error between the classification value and the annotation information of the detection box, and λ represents the weight of the detection branch of the original model.

[0053] In this embodiment, the detection error L of the target model is obtained. student , where L student =L reg +λ1L cls Lreg is the error between the detection box regression value and the annotation information, Lcls is the error between the detection box classification value and the annotation information, and λ1 represents the weight of the target model branch.

[0054] In this embodiment, the classification feature maps of the original model and the target model are passed through activation layers respectively, and the root mean square error (RMSE) is calculated, denoted as Lhm. The absolute error values ​​Lwlh and Lxyz of the two models are calculated based on the size and center point detection heads of the original and target models. Based on the absolute error values ​​Lwlh and Lxyz of the original and target models, the distillation error L between the two models is generated. kd =L hm +λ2L xyz +λ3L wlh .

[0055] In this embodiment, during the multiple rounds of iterative training of the original model, the target model is simultaneously subjected to multiple rounds of iterative training based on the distillation error, thereby obtaining the overall error L of the target model. 总体误差 =L student +λ4L kd λ4 represents the weight of the branch between the two models.

[0056] Optionally, in this embodiment, when the overall error of the target model decreases to the target value, the model accuracy of the target model is verified. If the model accuracy reaches the target accuracy value, a higher-precision 3D point cloud target detection student model is obtained.

[0057] This embodiment acquires 3D point cloud data of the driving road during the operation of an autonomous vehicle. The autonomous vehicle is equipped with a low-power embedded platform and uses an installed LiDAR sensor to collect the 3D point cloud data of the driving road. The 3D point cloud data is then input into an optimized model of a 3D point cloud target detection model to identify at least one target located within the area of ​​the driving road. The optimized model of the 3D point cloud target detection model runs on the low-power embedded platform. In other words, this application, based on a low-power embedded platform, uses an installed LiDAR sensor to collect 3D point cloud data of the driving road during the operation of an autonomous vehicle. By inputting this data into an optimized model of a 3D point cloud target detection model on the low-power embedded platform, at least one target located within the area of ​​the driving road can be identified. This expands the application range of the 3D target detection model and solves the technical problem of its limited applicability.

[0058] Example 2

[0059] The technical solutions of the embodiments of the present invention will be illustrated below with reference to preferred embodiments.

[0060] Artificial intelligence chips are one of the core technologies of the AI ​​era, determining the hardware and software infrastructure and development ecosystem of autonomous driving computing platforms. To ensure that autonomous vehicles can make correct judgments in different scenarios, they need to achieve real-time dynamic acquisition and recognition of surrounding environmental information. This information includes, but is not limited to, the vehicle's status, traffic flow information, road conditions, and traffic signs, to meet the needs of the vehicle's decision-making system. To meet the needs of environmental perception, autonomous vehicles are often equipped with LiDAR sensors to acquire real-time 3D point cloud data of the road. This device can be used to input the point cloud data into a 3D point cloud object detection algorithm. The deep learning-based 3D point cloud object detection algorithm realizes end-to-end perception capabilities, thereby obtaining various target information of the surrounding environment and providing accurate information for the environmental perception of autonomous vehicles.

[0061] In LiDAR sensors, general-purpose chips such as graphics processing units (GPUs) still dominate. However, while GPUs are versatile, fast, and efficient, they also consume a lot of power, making them unsuitable for autonomous vehicles. This has led to the development of AI processors specifically designed for autonomous driving. Although these AI processors offer high computing power, high performance, and low power consumption, their complex design places high demands on model portability. Even though many high-precision 3D object detection models have emerged, limitations in computing power, portability, and accuracy often prevent their widespread adoption in autonomous vehicles, resulting in the technical challenge of limited applicability of 3D object detection models.

[0062] Meanwhile, the 3D point cloud target detection algorithm running in autonomous vehicles has the following disadvantages: (1) Complex structure, low computing power platform, and high latency. In order to meet the detection frame rate, only a high computing power platform can be used, which increases the cost and power consumption; (2) Poor portability, some operators depend on specific hardware and can only be deployed on one or a few dedicated platforms; (3) In order to meet the accuracy requirements, floating-point precision calculation is used, which causes some embedded platforms designed for low-precision calculation to fail to fully utilize their computing power, further increasing the detection latency.

[0063] In one related technology, a voxelized 3D network and arithmetic mean (VFE) implementation algorithm based on an embedded FPGA platform can be implemented. The preprocessing algorithm for 3D point cloud target detection is implemented on the embedded platform. However, this algorithm is limited to the embedded FPGA platform and still has the problem of limited application range of the 3D target detection model.

[0064] Another related technique proposes to input point cloud feature maps into a 3D object detection model, train the 3D object detection model, and obtain a trained 3D object detection model to achieve object detection. However, this algorithm has the problem of difficulty in obtaining a high-precision, low-latency detection model.

[0065] To overcome the above problems, this embodiment designs a 3D point cloud target detection method that can run on low-power embedded platforms and achieve high accuracy and low latency. By trimming the high-computing-power, high-precision teacher model, a lightweight student model backbone network is obtained. Combined with lightweight point cloud feature preprocessing, a highly portable, low-computing-power student model is achieved. By distilling the teacher model, the accuracy of the student model reaches the same level as the teacher model. A fixed-point model can be obtained by adopting a quantized perception training method. Finally, a low-latency, high-precision 3D point cloud target detection model that can run on low-power embedded devices is realized, thereby improving the running speed while maintaining high accuracy. It also has strong portability on various neural network processing chips, thus solving the problem of the limited application range of 3D target detection models.

[0066] In this embodiment, the 3D point cloud target detection method consists of four parts: (1) Training the teacher model. A high-precision 3D target detection teacher model is constructed and trained. This teacher model has the ability to achieve high-performance radar perception. Due to limitations in computing power and operators, this teacher model cannot be directly transferred to the embedded platform, but through training, a high-precision template model can be achieved; (2) Tailoring the teacher model into a student model. The teacher model obtained in the first part is tailored so that the lightweight model after tailoring meets the computing power requirements of the embedded platform; (3) Training the student model. That is, the feature extraction capability of the teacher model is transferred to the student model through knowledge distillation transfer learning, so that the student model absorbs the detection capability of the teacher model. The guidance learning of the teacher model improves the generalization of the student network and helps to accelerate convergence. Finally, a student model with the same high precision is obtained; (4) Performing fixed-point quantization on the student model. To further compress the student model, it is quantized into a fixed-point model with almost no loss of accuracy. Specifically, the continuously or discretely taking floating-point parameters (weights or tensors) in the network are linearly mapped to discrete values ​​approximating fixed-point values, replacing the original floating-point data, while maintaining the input and output as floating-point types. This achieves the goals of reducing model size, reducing model memory consumption, and accelerating model inference speed. The method is further described below.

[0067] Step 1: Construct a 3D point cloud target detection teacher model by combining 3D sparse convolutional coding, a heavyweight backbone network, and multiple detection heads.

[0068] Step 2, set the loss function for the teacher model. Where L reg To detect the error between the bounding box regression value and the annotation information, L cls The error is calculated as L, representing the error between the bounding box classification value and the annotation information. teacher =L reg +λL clsThe teacher model is trained using stochastic gradient descent. After multiple rounds of iterative training, training stops when the overall error decreases to a certain level. By validating the accuracy of the teacher model, a high-precision 3D point cloud object detection teacher model is obtained when the required accuracy is achieved.

[0069] Step 3: Use the pruning algorithm to trim the heavyweight backbone network in the teacher model to obtain the lightweight backbone network in the student model.

[0070] Step 4: By combining the point cloud raster preprocessing of the bird's-eye view, the above-mentioned cropped lightweight backbone network and detection head are used to construct a 3D point cloud target detection student model.

[0071] Step 5, similar to the teacher model in Step 2, the detection error of the student model is L. student =L reg +λ1L cls Input the same batch of data and obtain the outputs of the teacher model and the student model respectively. Pass the classification feature maps output by the two models through activation layers and calculate the root mean square error, denoted as L. hm The absolute error of the detection head outputting the dimensions and center points of the two models is calculated, and L is obtained respectively. wlh L xyz Thus, the distillation error is L. kd =L hm +λ2L xyz +λ3L wlh Thus, the overall error L of the student model is obtained. 总体误差 =L student +λ4L kd ;

[0072] Step 6: Train the student model using stochastic gradient descent. After multiple rounds of iterative training, when the overall error of the student model decreases to a certain level, verify the accuracy of the student model. When the accuracy meets the requirements, a 3D point cloud target detection student model with the same high accuracy can be obtained.

[0073] Step 7: Use the quantization perception training method to perform fixed-point compression on the student model, and finally output a 3D point cloud target detection model that can run on a low-power embedded device.

[0074] This embodiment acquires 3D point cloud data of the driving road during the operation of an autonomous vehicle. The autonomous vehicle is equipped with a low-power embedded platform and uses an installed LiDAR sensor to collect the 3D point cloud data of the driving road. The 3D point cloud data is then input into an optimized model of a 3D point cloud target detection model to identify at least one target located within the area of ​​the driving road. The optimized model of the 3D point cloud target detection model runs on the low-power embedded platform. In other words, this application, based on a low-power embedded platform, uses an installed LiDAR sensor to collect 3D point cloud data of the driving road during the operation of an autonomous vehicle. By inputting this data into an optimized model of a 3D point cloud target detection model on the low-power embedded platform, at least one target located within the area of ​​the driving road can be identified. This expands the application range of the 3D target detection model and solves the technical problem of its limited applicability.

[0075] Example 3

[0076] According to an embodiment of the present invention, a control device for a 3D point cloud target detection method is also provided. It should be noted that this 3D point cloud target detection device can be used to execute the 3D point cloud target detection method in Embodiment 1.

[0077] Figure 2 This is a schematic diagram of a 3D point cloud target detection device according to an embodiment of the present invention. Figure 2 As shown, the 3D point cloud target detection device 200 may include: a first acquisition module 201 and an identification module 202.

[0078] The first acquisition module 201 is used to acquire three-dimensional point cloud data of the driving road during the driving process of the autonomous vehicle. The autonomous vehicle is equipped with a low-power embedded platform and acquires the three-dimensional point cloud data of the driving road through the installed LiDAR sensor.

[0079] The recognition module 202 is used to input the three-dimensional point cloud data of the driving road into the optimized model of the 3D point cloud target detection model, and to identify at least one target information located within the area where the driving road is located; wherein, the optimized model of the 3D point cloud target detection model is a model running on a low-power embedded platform.

[0080] Optionally, the second acquisition module is used to acquire the original model of the 3D point cloud target detection model.

[0081] Optionally, a pruning module is used to obtain a backbone network model by pruning the original model.

[0082] Optionally, a preprocessing module is used to preprocess the backbone network model by combining lightweight point cloud features to generate the target model.

[0083] Optionally, a generation module is used to generate the overall error of the target model based on the detection error of the original model and the detection error of the target model.

[0084] Optionally, the pruning module includes a sub-pruning module, which is used to prune the heavyweight backbone network part in the original model using a pruning algorithm to obtain a backbone network model, wherein the backbone network model is the lightweight backbone network in the target model.

[0085] Optionally, the training module is used to perform multiple rounds of iterative training on the original model based on a preset loss function. When the model accuracy of the original model reaches the target accuracy value after multiple rounds of iterative training, an optimized original model is generated, wherein the loss function is a function based on the stochastic gradient descent algorithm.

[0086] Optionally, the generation module includes a first generation unit, which is used to verify the model accuracy of the original model if the overall error of the target model decreases to the target value after multiple rounds of iterative training of the original model; if the model accuracy reaches the target accuracy value, it is determined that the accuracy of the target model also reaches the target accuracy value; and to perform fixed-point compression on the target model that has reached the target accuracy value to generate an optimized model of the 3D point cloud target detection model.

[0087] Optionally, the second acquisition module includes a first acquisition unit, used to acquire the original model detection error L. teacher , where L teacher =L reg +λL cls L reg To detect the error between the bounding box regression value and the annotation information, L cls To detect the error between the bounding box classification value and the annotation information, λ represents the weight of the detection branch of the original model.

[0088] Optionally, the second acquisition module includes a second acquisition unit for acquiring the detection error L of the target model. student , where L student =L reg +λ1L cls L reg To detect the error between the bounding box regression value and the annotation information, L cls To detect errors in bounding box classification values ​​and annotation information, λ1 represents the weight of the target model branch.

[0089] Optionally, the generation module includes a second generation unit, used to pass the classification feature maps of the original model and the target model through activation layers to calculate the root mean square error L. hmThe absolute error L between the original model and the target model is calculated by using the detection head to measure the dimensions and center points of the two models. wlh and L xyz Based on the root mean square error, the absolute value of the error between the original model and the target model L is calculated. wlh and L xyz Generate the distillation error L between the two models. k d = L hm +λ2L xyz +λ3L wlh During the multiple rounds of iterative training of the original model, based on the distillation error, the target model is simultaneously subjected to multiple rounds of iterative training to obtain the overall error of the target model. When the overall error of the target model decreases to the target value, if the model accuracy reaches the target accuracy value, it is determined that the accuracy of the target model has also reached the target accuracy value.

[0090] In this embodiment of the invention, during the operation of an autonomous vehicle, three-dimensional point cloud data of the driving road is acquired. The autonomous vehicle is equipped with a low-power embedded platform, and the three-dimensional point cloud data of the driving road is collected by an installed LiDAR sensor. The three-dimensional point cloud data of the driving road is input into an optimized model of a 3D point cloud target detection model to identify at least one target located within the area of ​​the driving road. The optimized model of the 3D point cloud target detection model runs on the low-power embedded platform. In other words, this application, based on a low-power embedded platform, acquires three-dimensional point cloud data of the driving road through an installed LiDAR sensor during the operation of an autonomous vehicle. By inputting this data into an optimized model of a 3D point cloud target detection model on the low-power embedded platform, at least one target located within the area of ​​the driving road is identified. This achieves the technical effect of expanding the application range of the 3D target detection model and solves the technical problem of the limited application range of the 3D target detection model.

[0091] Example 4

[0092] According to an embodiment of the present invention, a computer-readable storage medium is also provided, the storage medium including a stored program, wherein the program executes the 3D point cloud target detection method described in Embodiment 1.

[0093] Example 5

[0094] According to an embodiment of the present invention, a processor is also provided for running a program, wherein the program executes the 3D point cloud target detection method described in Embodiment 1.

[0095] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0096] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0097] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0098] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0099] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0100] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0101] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting targets in 3D point clouds, characterized in that, Applied to autonomous vehicles, the method includes: During the operation of an autonomous vehicle, three-dimensional point cloud data of the driving road is acquired. The autonomous vehicle is equipped with a low-power embedded platform and acquires the three-dimensional point cloud data of the driving road through an installed lidar sensor. The three-dimensional point cloud data of the driving road is input into the optimization model of the 3D point cloud target detection model to identify at least one target information located within the area where the driving road is located; The optimized model of the 3D point cloud target detection model is a model running on the low-power embedded platform. The method further includes: obtaining the original model of the 3D point cloud target detection model, wherein the original model includes a combination of 3D sparse convolutional coding, a heavyweight backbone network, and multiple detection heads; the 3D sparse convolutional coding is used to reconstruct information coding from sparse and irregular data; the heavyweight backbone network is used to determine the backbone network model; the detection head includes a classification branch and a regression branch; the classification branch is used to represent the type of detection information; the regression branch is used to represent the features of the detection information; the features of the detection information include at least one of the following: length, width, height, depth, and orientation angle; cropping the original model to obtain the backbone network model; preprocessing the backbone network model based on lightweight point cloud features by combining point cloud grids from a bird's-eye view to generate a target model; and generating the overall error of the target model based on the detection error of the original model and the detection error of the target model. Before inputting the 3D point cloud data of the driving road into the optimized model of the 3D point cloud target detection model, the method further includes: performing multiple rounds of iterative training on the original model; verifying the model accuracy of the original model if the overall error of the target model decreases to the target value; determining that the accuracy of the target model has reached the target accuracy value if the model accuracy reaches the target accuracy value; and using a quantization perception training strategy to perform fixed-point compression on the target model that has reached the target accuracy value to generate an optimized model of the 3D point cloud target detection model running on the low-power embedded platform. Based on the detection error of the original model and the detection error of the target model, the overall error of the target model is generated, including: obtaining the detection error of the original model. , wherein , Used to represent the error between the detection box regression value and the annotation information. To detect errors in the bounding box classification values ​​and annotation information, The weights of the detection branch of the original model are represented; the detection error of the target model is obtained. , wherein , To detect the error between the bounding box regression value and the annotation information, To detect errors in the bounding box classification values ​​and annotation information, The weights of the branches of the target model are represented; the classification feature maps of the original model and the target model are passed through activation layers respectively, and the root mean square error is calculated. The detection head, based on the dimensions and center points of the original model and the target model, calculates the absolute value of the error between the two models. and Based on the root mean square error, the absolute value of the error between the original model and the target model is calculated. and Generate distillation error between the two models During the multiple rounds of iterative training of the original model, based on the distillation error, the target model is simultaneously subjected to multiple rounds of iterative training to obtain the overall error of the target model. When the overall error of the target model decreases to the target value, if the model accuracy reaches the target accuracy value, it is determined that the accuracy of the target model has also reached the target accuracy value.

2. The method according to claim 1, characterized in that, The heavyweight backbone network portion of the original model is pruned using a pruning algorithm to obtain the backbone network model, wherein the backbone network model is the lightweight backbone network in the target model.

3. The method according to claim 1, characterized in that, The original model is trained iteratively for multiple rounds based on a preset loss function. When the model accuracy of the original model reaches the target accuracy value after multiple rounds of iterative training, an optimized original model is generated. The loss function is a function based on the stochastic gradient descent algorithm.

4. A device for detecting 3D point cloud targets, characterized in that, The device, intended for use in autonomous vehicles, includes: The first acquisition module is used to acquire three-dimensional point cloud data of the driving road during the driving process of the autonomous vehicle. The autonomous vehicle is equipped with a low-power embedded platform and acquires the three-dimensional point cloud data of the driving road through the installed lidar sensor. The identification module is used to input the three-dimensional point cloud data of the driving road into the optimization model of the 3D point cloud target detection model, and identify at least one target information located within the area where the driving road is located; The optimized model of the 3D point cloud target detection model is a model running on the low-power embedded platform. The device is further configured to perform the following steps: obtaining the original model of the 3D point cloud target detection model, wherein the original model includes a combined 3D sparse convolutional coding, a heavyweight backbone network, and multiple detection heads, the 3D sparse convolutional coding being used to reconstruct information encoding from sparse and irregular data, the heavyweight backbone network being used to determine the backbone network model, the detection head including a classification branch and a regression branch, the classification branch being used to represent the type of detection information, the regression branch being used to represent the features of the detection information, the features of the detection information including at least one of the following: length, width, height, depth, and orientation angle; cropping the original model to obtain the backbone network model; preprocessing the backbone network model based on lightweight point cloud features by combining point cloud grids from a bird's-eye view to generate a target model; and generating the overall error of the target model based on the detection error of the original model and the detection error of the target model. The original model is trained iteratively for multiple rounds; if the overall error of the target model decreases to the target value, the accuracy of the original model is verified; if the model accuracy reaches the target accuracy value, it is determined that the accuracy of the target model has reached the target accuracy value; using a quantization perception training strategy, the target model that has reached the target accuracy value is compressed at fixed points to generate an optimized model of the 3D point cloud target detection model running on the low-power embedded platform; The device is also configured to perform the following step: acquiring the original model detection error. , wherein , Used to represent the error between the detection box regression value and the annotation information. To detect errors in bounding box classification values ​​and annotation information, The weights of the detection branch of the original model are represented; the detection error of the target model is obtained. , wherein , To detect the error between the bounding box regression value and the annotation information, To detect errors in bounding box classification values ​​and annotation information, The weights of the branches of the target model are represented; the classification feature maps of the original model and the target model are passed through activation layers respectively, and the root mean square error is calculated. The detection head, based on the dimensions and center points of the original model and the target model, calculates the absolute value of the error between the two models. and Based on the root mean square error, the absolute value of the error between the original model and the target model is calculated. and Generate distillation error between the two models During the multiple rounds of iterative training of the original model, based on the distillation error, the target model is simultaneously subjected to multiple rounds of iterative training to obtain the overall error of the target model. When the overall error of the target model decreases to the target value, if the model accuracy reaches the target accuracy value, it is determined that the accuracy of the target model has also reached the target accuracy value.

5. The apparatus according to claim 4, characterized in that, The device further includes: The second acquisition module is used to acquire the original model of the 3D point cloud object detection model; The pruning module is used to obtain the backbone network model by pruning the original model; The preprocessing module is used to preprocess the backbone network model by combining lightweight point cloud features to generate the target model; The generation module is used to generate the overall error of the target model based on the detection error of the original model and the detection error of the target model.

6. The apparatus according to claim 5, characterized in that, The pruning module includes a sub-pruning module, used to prune the heavyweight backbone network portion of the original model using a pruning algorithm to obtain the backbone network model, wherein the backbone network model is the lightweight backbone network in the target model.

7. The apparatus according to claim 5, characterized in that, The device further includes a training module, used to perform multiple rounds of iterative training on the original model based on a preset loss function, and to generate an optimized original model when the model accuracy of the original model reaches the target accuracy value after the multiple rounds of iterative training, wherein the loss function is a function based on the stochastic gradient descent algorithm.

Images