Point cloud target detection method, device, equipment, medium and product

Abstract

The invention discloses a point cloud target detection method, device and equipment, a medium and a product, and belongs to the technical field of deep learning. The method comprises the following steps: acquiring point cloud data collected by a laser radar sensor; the point cloud data comprises a plurality of points, and an x-axis coordinate value, a y-axis coordinate value, a z-axis coordinate value and reflection intensity of each point; determining a plurality of point cloud columns corresponding to the to-be-detected area according to the point cloud data; each point cloud column comprises at least one point in a plurality of points; for each point cloud column, according to the point cloud density of the point cloud column, the maximum value, the mean value and the minimum value of the x-axis coordinate values, the y-axis coordinate values and the z-axis coordinate values of all the points in the point cloud column, and the mean value and the maximum value of the reflection intensity of all the points in the point cloud column, determining a target pseudo image of the to-be-detected area; and inputting the target pseudo image into the trained target detection network to obtain a target detection result of the to-be-detected area. Therefore, the detection precision of the target detection network can be improved.

Description

Technical Field

[0001] The present application relates to the field of deep learning, and in particular to a method, device, equipment, medium and product for point cloud target detection. Background Art

[0002] With the rapid development of deep learning algorithms and sensor devices, 3D object detection technology based on Lidar point clouds has received widespread attention in the field of autonomous driving. Compared with RGB images, Lidar 3D point clouds are insensitive to lighting changes and directly contain the geometric position information of objects in the scene.

[0003] In the prior art, there are two main technical solutions for Lidar-based 3D target detection. One is a voxel-based perception algorithm, which first converts the input irregular point cloud data into a regular 3D grid, and then uses 3D convolution to extract features. In order to further improve computational efficiency, 3D sparse convolution is usually used to skip convolution calculations on empty grids. However, for most hardware platforms, it is difficult to support the deployment and quantization of 3D convolution.

[0004] The other type is a pillar-based perception algorithm, which first converts the point cloud into a point cloud pillar, that is, ignores the z value and only voxelizes it on the xoy plane. Then, a fully connected layer is used to learn the features of each point inside the pillar. After that, a neural network is used to learn these features, and finally the target detection task based on the Lidar3D point cloud is achieved. Although this method has a fast detection speed, it usually uses maximum pooling to aggregate the features of the points inside the pillar, resulting in a lot of local information loss, making the final detection effect unsatisfactory, especially for small targets such as pedestrians and cyclists. Therefore, the existing technology has the problem of low detection accuracy. Summary of the invention

[0005] The embodiments of the present application provide a method, apparatus, device, medium and product for point cloud target detection, which can improve the accuracy of point cloud target detection.

[0006] In a first aspect, an embodiment of the present application provides a method for point cloud target detection, the method comprising:

[0007] Obtain point cloud data collected by the laser radar sensor; the point cloud data includes multiple points, as well as the x-axis coordinate value, y-axis coordinate value, z-axis coordinate value and reflection intensity of each point;

[0008] Determine a plurality of point cloud columns corresponding to the area to be detected according to the point cloud data; each point cloud column includes at least one point among the plurality of points;

[0009] For each point cloud column, determine the target pseudo image of the area to be detected according to the point cloud density of the point cloud column, the maximum, mean and minimum values ​​of the x-axis coordinate values, y-axis coordinate values ​​and z-axis coordinate values ​​of all the points in the point cloud column, and the mean and maximum values ​​of the reflection intensity of all the points in the point cloud column;

[0010] The target pseudo image is input into the trained target detection network to obtain the target detection result of the area to be detected.

[0011] In a second aspect, an embodiment of the present application provides a point cloud target detection device, the device comprising:

[0012] The acquisition module is used to acquire the point cloud data collected by the laser radar sensor; the point cloud data includes multiple points, as well as the x-axis coordinate value, y-axis coordinate value, z-axis coordinate value and reflection intensity of each point;

[0013] A point cloud column determination module is used to determine a plurality of point cloud columns corresponding to the area to be detected according to the point cloud data; each point cloud column includes at least one point among the plurality of points;

[0014] a pseudo image determination module, for determining, for each point cloud column, a target pseudo image of the area to be detected according to the point cloud density of the point cloud column, the maximum, average and minimum values ​​of the x-axis coordinate values, y-axis coordinate values ​​and z-axis coordinate values ​​of all the points in the point cloud column, and the average and maximum values ​​of the reflection intensity of all the points in the point cloud column;

[0015] The detection module is used to input the target pseudo image into the trained target detection network to obtain the target detection result of the area to be detected.

[0016] In a third aspect, an embodiment of the present application provides a point cloud target detection device, the device comprising:

[0017] a processor and a memory storing computer program instructions;

[0018] When the processor executes the computer program instructions, the method for point cloud target detection as described in the first aspect is implemented.

[0019] In a fourth aspect, an embodiment of the present application provides a computer storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method for point cloud target detection as described in the first aspect is implemented.

[0020] In a fifth aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the method for point cloud target detection as described in the first aspect.

[0021] In the embodiment of the present application, the point cloud data collected by the laser radar sensor is first obtained, and the point cloud data includes multiple points, and the x-axis coordinate value, y-axis coordinate value, z-axis coordinate value and reflection intensity of each point, and then multiple point cloud columns corresponding to the area to be detected are determined according to the point cloud data, and each point cloud column includes at least one point among the multiple points. For each point cloud column, according to the point cloud density of the point cloud column, the maximum, mean and minimum values ​​of the x-axis coordinate value, y-axis coordinate value and z-axis coordinate value of all points in the point cloud column, and the mean and maximum values ​​of the reflection intensity of all points in the point cloud column, the pseudo image of the area to be detected is determined, and finally the pseudo image is input into the trained target detection network to obtain the target detection result of the area to be detected. In this way, the manually designed features, namely the maximum, mean and minimum values ​​of the x-axis coordinate values, y-axis coordinate values ​​and z-axis coordinate values ​​of all points, as well as the mean and maximum values ​​of the reflection intensity of all points in the point cloud column, are directly combined into 12 input channels to generate a pseudo image as the input of the subsequent detection network, which minimizes the information loss of the original point cloud after dimensionality reduction, retains the geometric features of the original point cloud, and improves the detection accuracy of the detection network. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] Figure 1 is a flow chart of a method for point cloud target detection provided by an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of a point cloud pre-processing provided in an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of the structure of a target detection network provided in an embodiment of the present application;

[0026] Figure 4 It is a structural schematic diagram of a point cloud target detection device provided in an embodiment of the present application;

[0027] Figure 5 It is a schematic diagram of the hardware structure of a point cloud target detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0029] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0030] The following is a detailed description of the xx method provided in the embodiment of the present application through some embodiments and their application scenarios in combination with the accompanying drawings.

[0031] See also Figure 1 , Figure 1 : is a flow chart of a method for point cloud target detection provided by an embodiment of the present application. Figure 1 As shown, the method for point cloud target detection may include the following steps:

[0032] Step 101: Obtain point cloud data collected by a laser radar sensor; the point cloud data includes multiple points, as well as the x-axis coordinate value, y-axis coordinate value, z-axis coordinate value and reflection intensity of each point.

[0033] LiDAR sensors are one of the core devices for autonomous vehicles to perceive the environment. They generate high-precision three-dimensional point cloud images by emitting laser pulses and receiving reflected signals. Based on the generated three-dimensional point cloud images, autonomous vehicles can identify objects in the surrounding environment, including pedestrians, vehicles, buildings, etc., to ensure driving safety.

[0034] In an embodiment of the present application, the point cloud data collected by the laser radar contains n target points, as well as information in four dimensions, namely, the x-axis coordinate value, y-axis coordinate value, z-axis coordinate value, and reflection intensity of each target point, which can be represented by a (n, 4) tensor.

[0035] Step 102: Determine a plurality of point cloud columns corresponding to the area to be detected according to the point cloud data; each point cloud column includes at least one point among the plurality of points.

[0036] In an embodiment of the present application, before the collected point cloud data is input into a target detection network for target detection, the collected point cloud data may be pre-processed.

[0037] See also Figure 2 , Figure 2 Schematic diagram of a point cloud pre-processing provided by an embodiment of the present application. Figure 2 As shown, first, the point cloud data of the area to be detected is screened out, and the area to be detected is a specific 3D region of interest in the x-axis, y-axis, and z-axis directions with the laser radar sensor as the coordinate origin. Then, according to the preset size, the area to be detected is divided into multiple cylinders, where the cylinder size is the actual size corresponding to each pixel of the BEV pseudo image (Bird's Eye View) that is compatible in size with the training data of the trained target detection network.

[0038] Each cylinder retains a maximum of N points through random sampling. If it is less than N, zero filling is used to retain only non-empty cylinders as point cloud columns, and finally a tensor of (P, N, D) dimensions is obtained. In one example, N can be set to 64. Among them, D represents the 3D position of each point, that is, the x-axis coordinate value, y-axis coordinate value, z-axis coordinate value and point cloud reflection intensity r, a total of 4 dimensions, that is, D = 4; P is the number of non-empty point cloud columns in each area to be detected, and the value is not fixed.

[0039] Step 103: For each point cloud column, determine the target pseudo image of the area to be detected based on the point cloud density of the point cloud column, the maximum, mean and minimum values ​​of the x-axis coordinate values, y-axis coordinate values ​​and z-axis coordinate values ​​of all points in the point cloud column, and the mean and maximum values ​​of the reflection intensity of all points in the point cloud column.

[0040] like Figure 2As shown, in an embodiment of the present application, from a deployment-friendly perspective, the manually designed features, including: the mean and maximum values ​​of x, y, z, and r, the minimum values ​​of x, y, and z, and the point cloud density, are directly combined into 12 input channels, and then mapped back to the corresponding positions of the original BEV grid according to the index corresponding to the point cloud column, and finally the BEV pseudo image is obtained as the subsequent input. The obtained BEV pseudo image can be represented by a tensor with a dimension of (Batch, C, H, W). Among them, C represents the number of channels of the input feature, and in the embodiment of the present application, C=12. Batch (batch size) refers to the number of data samples processed at one time. The size of Batch can be set according to the specific task and data set. H (height) refers to the height of the image, in pixels. W (width) refers to the width of the image, also in pixels.

[0041] In this way, there is no need to encode the point cloud column, avoiding the problem that some operators are not supported in the point cloud feature encoding link for some specific embedded chips. At the same time, the information loss after the original point cloud dimensionality reduction is minimized, and the geometric features of the original point cloud are retained.

[0042] Step 104: Input the target pseudo image into the trained target detection network to obtain the target detection result of the area to be detected.

[0043] See also Figure 3 , Figure 3 Schematic diagram of a target detection network provided in an embodiment of the present application. Figure 3 As shown, the backbone network part of the target detection network adopts the RegNet network, and the neck network part adopts the FPN (feature pyramid network) structure.

[0044] RegNet is a network structure obtained by searching the neural network design space, which has good performance and generalization ability. After the pseudo image is input into the trained target detection network, the multi-scale features of the pseudo image are first extracted through the RegNet network, and then the extracted multi-scale features are fused through the feature pyramid network to obtain the feature map of the area to be detected. The feature pyramid network in the embodiment of the present application adopts top-down feature layers and lateral connections, which can provide multi-scale feature representation, which is helpful for detecting targets of different sizes.

[0045] After obtaining the feature map of the area to be detected, the detection head added to the selected layer of the feature pyramid network detects the feature map and outputs the target detection result of the area to be detected. The detection result includes the category information and location information of the target contained in the area to be detected. In the field of autonomous driving, the category information can include pedestrians, cyclists, cars, trucks, etc.

[0046] The point cloud target detection method of this embodiment first obtains point cloud data collected by a laser radar sensor, the point cloud data includes multiple points, and the x-axis coordinate value, y-axis coordinate value, z-axis coordinate value and reflection intensity of each point, and then determines multiple point cloud columns corresponding to the area to be detected based on the point cloud data, each point cloud column includes at least one point among the multiple points. For each point cloud column, a pseudo image of the area to be detected is determined based on the point cloud density of the point cloud column, the maximum, mean and minimum values ​​of the x-axis coordinate value, y-axis coordinate value and z-axis coordinate value of all points in the point cloud column, and the mean and maximum values ​​of the reflection intensity of all points in the point cloud column, and finally the pseudo image is input into the trained target detection network to obtain the target detection result of the area to be detected. In this way, the manually designed features, namely the maximum, mean and minimum values ​​of the x-axis coordinate values, y-axis coordinate values ​​and z-axis coordinate values ​​of all points, as well as the mean and maximum values ​​of the reflection intensity of all points in the point cloud column, are directly combined into 12 input channels to generate a pseudo image as the input of the subsequent detection network, which minimizes the information loss of the original point cloud after dimensionality reduction, retains the geometric features of the original point cloud, and improves the detection accuracy of the detection network.

[0047] In some embodiments, before inputting the pseudo image into the trained object detection network to obtain the object detection result of the area to be detected, the method may further include:

[0048] Obtain a training sample set; the training sample set includes multiple training samples, each training sample includes a training pseudo image and annotation information corresponding to the training pseudo image, the training pseudo image is obtained based on point cloud data, the annotation information includes category information and position information of the target in the training pseudo image, and the position information of the target includes a true value box of the target;

[0049] Using the training sample set to train the target detection network to be trained until the training stop condition is met, thereby obtaining a trained target detection network;

[0050] Wherein, when using the i-th training sample in the training sample set to train the target detection network to be trained, the following steps are included:

[0051] Input the i-th training sample into the target detection network to be trained to obtain a feature map corresponding to the i-th training sample; the feature map includes at least one anchor point box;

[0052] Calculate the intersection-over-union ratio between the anchor box and the true value box;

[0053] The first k anchor boxes with the highest intersection-over-union ratio with the true value box are determined as the positive samples corresponding to the true value box, where k is a positive integer;

[0054] The category information and position information corresponding to the true value frame are determined as the category information and position information of the positive sample corresponding to the true value frame;

[0055] Based on the positive samples, determine the total loss value of the i-th training sample;

[0056] When the total loss value does not meet the training stop condition, the model parameters of the target detection network are adjusted, and the adjusted target detection network is trained using the i+1th training sample in the training sample set until the total loss value of the target detection network meets the training stop condition, thereby obtaining a trained target detection network, where i is a positive integer.

[0057] Specifically, before using the target detection network to perform point cloud target detection, the target detection network can be trained first. In the training stage, a training sample set is first obtained. Each training sample in the training sample set includes a training pseudo image and annotation information corresponding to the training pseudo image. The annotation information includes the category information and position information of the target in the training pseudo image. The position information of the target includes the true value box of the target. It should be noted that the training pseudo image as a training sample is obtained by pre-processing the point cloud data. The target detection network to be trained is trained using the training sample set until the training stop condition is met, thereby obtaining a trained target detection network.

[0058] For each training sample in the training sample set, after it is input into the target detection network to be trained, the RegNet network and the feature pyramid network of the target detection network to be trained can be used to extract and fuse features in turn to generate the corresponding feature map. Then, based on the feature map, the allocation of positive and negative samples is achieved through the dynamic anchor matching strategy. That is, for the anchor box (anchor) corresponding to each feature point on the feature map, the intersection over Union (IoU) of each anchor and each true value box of the feature map is calculated respectively. It should be noted that the number of anchor boxes and true value boxes is at least one, and the number of anchor boxes is greater than the true value box. In target detection, the intersection over Union ratio is usually used to measure the degree of overlap between the predicted bounding box and the true bounding box, that is, the true value box.

[0059] For each true value box, the first k anchor boxes with the highest intersection-over-union ratio are determined as positive samples, and then the category information and position information corresponding to the true value box are determined as the category information and position information of the positive sample corresponding to the true value box. In other words, for any true value box, there is at least one corresponding anchor box, and the category information and position information of the corresponding anchor box are the category information and position information corresponding to the arbitrary true value box.

[0060] Finally, the positive samples are detected by the detection head, and the total loss value of the i-th training sample is calculated based on the detection results. When the total loss value does not meet the training stop condition, the parameter value of the target detection network is updated by gradient back propagation, and the target detection network with updated parameters is trained using the i+1-th training sample in the training sample set until the total loss value of the target detection network meets the training stop condition, and a trained target detection network is obtained, where i is a positive integer. The stop condition is used to determine when to stop the training process to avoid overfitting, save computing resources, or achieve the expected model performance. In one example, the stop condition is that the total loss value of the i-th training sample is less than a preset threshold.

[0061] In some implementations, the trained object detection network includes a detection head, and determining the total loss value of the i-th training sample based on the positive sample may include:

[0062] The classification branch of the detection head outputs the classification confidence of each anchor box in the positive sample;

[0063] The regression branch of the detection head outputs the offset of each anchor box in the positive sample to the true value box corresponding to the positive sample;

[0064] Calculate the prediction quality of each anchor box in the positive sample based on the classification confidence and offset;

[0065] Determine the total loss value of the i-th training sample based on the prediction quality.

[0066] Specifically, the detection head of the trained object detection network includes a classification branch and a regression branch. For each anchor box in the positive sample, the classification branch is used to output the category information and classification confidence of the anchor box, and the regression branch is used to output the offset from the anchor box to the true value box.

[0067] Among them, the offset from the anchor box to the true value box includes the center point coordinate offset dx, dy, dz, the length, width and height scaling coefficients dl, dw, dh and the heading angle θ. Based on the offset, the 3D prediction box of each anchor box can be decoded. The intersection and union ratios of the prediction boxes corresponding to all anchor boxes and all true value boxes are calculated respectively, and then the prediction box detection quality is calculated according to the maximum value IoU_max of all intersection and union ratios. The calculation formula of the prediction box detection quality is as follows:

[0068] P IoU =max{min{(IoU-IoU_thr) / (max{IoU_max,IoU_thr}-IoU_thr+1e-6),0},1}

[0069] Among them, P IoUis the detection quality of the predicted box, IoU is the intersection over union (IoU) of the predicted box corresponding to the anchor box and the true value box, IoU_thr is the intersection over union (IoU) threshold, which can be set according to actual conditions, and IoU_max is the maximum value of all IoUs.

[0070] The calculation formula for the prediction quality of each anchor box in the positive sample is as follows:

[0071] q m =P cls *P IoU

[0072] Among them, q m is the prediction quality of the anchor box, P cls is the classification confidence, which can be calculated by the Sigmoid function based on the output result of the classification branch.

[0073] In the embodiment of the present application, the formula of the positive sample anchor loss function is as follows:

[0074]

[0075] Among them, L pos is the positive sample loss, q m is the prediction quality of the anchor box.

[0076] In the embodiment of the present application, the total loss value may include three parts: regression loss, classification loss and positive sample loss. According to the prediction quality of the anchor box, it can be determined whether a positive sample loss value is generated during this training process, thereby calculating the total loss value.

[0077] In one example, the regression loss is calculated using the Smooth L1 Loss function, and the classification loss is calculated using the FocalLoss function.

[0078] In some implementations, determining the total loss value of the i-th training sample according to the prediction quality of the anchor box corresponding to each positive sample may also include:

[0079] When the prediction quality of the anchor boxes in the positive samples is 0, the loss value of the i-th training sample is calculated according to the following formula:

[0080] L total =w1L box +w2L cls

[0081] Among them, L total is the total loss value, L box is the regression loss, w1 is the regression loss weight coefficient, L cls is the classification loss, and w2 is the classification loss weight coefficient.

[0082] Specifically, when the prediction quality of the anchor boxes in the positive samples is 0, it means that it is at the beginning of training, and the intersection-over-union ratio of the predicted box and the true value box of each positive sample anchor box is generally lower than the intersection-over-union ratio threshold, and no positive sample anchor loss is generated. Therefore, the total loss value of this training, that is, the i-th training sample, only includes regression loss and classification loss.

[0083] In some implementations, determining the total loss value of the i-th training sample according to the prediction quality may include:

[0084] When the prediction quality of the anchor box in the positive sample is greater than 0, the total loss value of the i-th training sample is calculated according to the following formula:

[0085]

[0086] Among them, L total is the total loss value, L box is the regression loss, w1 is the regression loss weight coefficient, L cls is the classification loss, w2 is the classification loss weight coefficient, L pos is the positive sample loss, w3 is the positive sample loss weight coefficient, q m is the prediction quality of the mth anchor box in the positive sample.

[0087] Specifically, when the prediction quality of the anchor box in the positive sample is greater than 0, it means that one or more anchor boxes whose IoU ratio between the predicted box and the true value box is greater than the IoU threshold begin to appear. This is the middle and late stages of training, and positive sample anchor loss is generated. Therefore, the total loss value of this training, that is, the i-th training sample, includes the positive sample anchor loss, regression loss, and classification loss.

[0088] In some embodiments, before training the target detection network to be trained using the training sample set until the training stop condition is met and the trained target detection network is obtained, the method further includes:

[0089] Obtain a pre-training dataset, which is a 2D image dataset;

[0090] The initial target detection network is trained using the pre-training dataset to obtain the target detection network to be trained.

[0091] Specifically, since the collection and annotation cost of 3D point cloud data is higher than that of 2D image data, the detection accuracy of the target detection model is low in the early stage when the accumulated amount of point cloud road data is small.

[0092] Based on this, in an embodiment of the present application, before using the training sample set to train the target detection network to be trained until the training stop condition is met and the trained target detection network is obtained, the initial target detection network is pre-trained using a 2D image data set to obtain the target detection network to be trained. Then, the target detection network to be trained is trained in combination with 3D point cloud data, so that the point cloud detection model can converge quickly and achieve higher accuracy in a short time. In one example, the initial target detection network is a RegNet network.

[0093] It should be noted that the various optional implementation modes introduced in the embodiments of the present application can be implemented in combination with each other or separately if they do not conflict with each other, and the embodiments of the present application are not limited to this.

[0094] For ease of understanding, a specific embodiment is used as an example for illustration:

[0095] Step 1: Get the original point cloud data with dimension (n, 4), where n represents the number of points and 4 represents the 3D position x, y, z and reflection intensity r of the point.

[0096] Step 2: According to the set effective range and the preset size of the point cloud column, different points are divided into different point cloud columns, and only non-empty point cloud columns are retained, and each point cloud column retains a maximum of 64 points, to obtain a tensor of (m, 64, 4) dimensions, where m represents the number of non-empty point cloud columns.

[0097] Step 3: Count the x-axis coordinates, y-axis coordinates, z-axis coordinates, mean and maximum values ​​of the reflection intensity r of all points in each non-empty point cloud column, the minimum values ​​of the x-axis coordinates, y-axis coordinates, z-axis coordinates, and the point cloud density, combine the above features into multiple channels, and map them back to the corresponding positions of the original BEV grid according to the index corresponding to the point cloud column, and finally obtain an input tensor of (Batch, C, H, W) dimensions, i.e., a pseudo image.

[0098] Step 4: Input the pseudo image into the target detection model, and output the target category information and location information. The target detection model uses the backbone network and neck network composed of 2D convolution to complete the feature extraction and feature fusion of the pseudo image to generate a feature map; the detection head part outputs the final detection result.

[0099] The same batch of point cloud data is pre-processed using the existing technology, and then tested using the detection model trained using the existing technology. The test results are evaluated respectively, and the evaluation results are shown in Table 1.

[0100] Table 1

[0101]

[0102]

[0103] The backbone network of the target detection network in the embodiment of the present application adopts RegNet, which is commonly used in visual detection tasks. The network has made a lot of optimizations in terms of the balance between accuracy and speed. It can fuse features of different scales, and fuse deep features that contain rich semantic information and have a larger receptive field with shallow features that focus on local areas.

[0104] In the process of training the target detection network in advance, a data balanced sampling strategy is adopted. For samples of categories with less frequent occurrence in the included data set, there is a higher probability that they will be added to the batch used for training, which improves the detection accuracy of categories with less frequent occurrence in the data set, thereby improving the overall detection accuracy of each category in practical applications.

[0105] In addition, traditional Lidar 3D point cloud detection networks mostly use random initialization for model training. Since the collection and annotation cost of 3D point cloud data is higher than that of 2D image data, the detection accuracy is low in the early stage when the accumulated amount of point cloud road data is small. During training, this solution first uses a large-scale 2D image dataset for pre-training as the initial weight loaded before training the target detection network, and then combines the 3D point cloud data for model fine-tuning, so that the point cloud detection model can converge quickly and achieve high accuracy in a short time.

[0106] According to Table 1, for each category of targets, the detection accuracy of the target detection network trained by the solution of the embodiment of the present application is higher than that of the prior art.

[0107] Furthermore, the detection head of the target detection network of the embodiment of the present application uses an improved dynamic anchor matching strategy, so that when detecting small target categories such as pedestrians and cyclists, the detection accuracy is significantly improved compared to the detection model trained with the prior art.

[0108] Based on the point cloud target detection method provided in the above embodiment, the present application also provides a specific implementation of the point cloud target detection device. Please refer to the following embodiment.

[0109] See also Figure 4 , Figure 4 Schematic diagram of the structure of a point cloud target detection device provided in an embodiment of the present application. Figure 4 As shown, the point cloud target detection device provided in the embodiment of the present application may include:

[0110] The acquisition module 401 is used to acquire point cloud data collected by the laser radar sensor; the point cloud data includes multiple points, and the x-axis coordinate value, y-axis coordinate value, z-axis coordinate value and reflection intensity of each point;

[0111] The point cloud column determination module 402 is used to determine a plurality of point cloud columns corresponding to the area to be detected according to the point cloud data; each point cloud column includes at least one point among the plurality of points;

[0112] The pseudo image determination module 403 is used to determine, for each point cloud column, a target pseudo image of the area to be detected according to the point cloud density of the point cloud column, the maximum, average and minimum values ​​of the x-axis coordinate values, y-axis coordinate values ​​and z-axis coordinate values ​​of all points in the point cloud column, and the average and maximum values ​​of the reflection intensity of all points in the point cloud column;

[0113] The detection module 404 is used to input the target pseudo image into the trained target detection network to obtain the target detection result of the area to be detected.

[0114] The point cloud target detection device provided in the embodiment of the present application can achieve Figure 1 The various processes implemented by the point cloud target detection device in the method embodiment will not be described again here to avoid repetition.

[0115] Figure 5 It is a schematic diagram of the hardware structure of a point cloud target detection device provided in an embodiment of the present application.

[0116] The point cloud object detection device may include a processor 501 and a memory 502 storing computer program instructions.

[0117] Specifically, the processor 501 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0118] The memory 502 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 502 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 502 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 502 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 502 is a non-volatile solid-state memory.

[0119] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of the present disclosure.

[0120] The processor 501 implements any one of the point cloud target detection methods in the above embodiments by reading and executing computer program instructions stored in the memory 502 .

[0121] In one example, the point cloud target detection device may further include a communication interface 503 and a bus 510. Figure 5 As shown, the processor 501, the memory 502, and the communication interface 503 are connected via a bus 510 and communicate with each other.

[0122] The communication interface 503 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0123] Bus 510 includes hardware, software or both, and the components of point cloud target detection equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-end bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnect (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 510 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0124] In addition, in combination with the point cloud target detection method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the point cloud target detection methods in the above embodiment is implemented.

[0125] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0126] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0127] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0128] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0129] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A method for point cloud target detection, characterized in that: include: Obtaining point cloud data collected by a laser radar sensor; the point cloud data includes a plurality of points, and an x-axis coordinate value, a y-axis coordinate value, a z-axis coordinate value, and a reflection intensity of each point; Determine a plurality of point cloud columns corresponding to the area to be detected according to the point cloud data; each of the point cloud columns includes at least one point among the plurality of points; For each of the point cloud columns, determine the target pseudo image of the area to be detected according to the point cloud density of the point cloud column, the maximum, mean and minimum values ​​of the x-axis coordinate values, y-axis coordinate values ​​and z-axis coordinate values ​​of all the points in the point cloud column, and the mean and maximum values ​​of the reflection intensity of all the points in the point cloud column; The target pseudo image is input into a trained target detection network to obtain a target detection result of the area to be detected.

2. The method according to claim 1, characterized in that Before inputting the pseudo image into a trained target detection network to obtain the target detection result of the area to be detected, the method further includes: Acquire a training sample set; the training sample set includes a plurality of training samples, each of the training samples includes a training pseudo image and annotation information corresponding to the training pseudo image, the training pseudo image is obtained based on point cloud data, the annotation information includes category information and position information of a target in the training pseudo image, and the position information of the target includes a true value box of the target; Using the training sample set to train the target detection network to be trained until a training stop condition is met, thereby obtaining the trained target detection network; Wherein, when the object detection network to be trained is trained using the i-th training sample in the training sample set, the following steps are included: Inputting the i-th training sample into the target detection network to be trained to obtain a feature map corresponding to the i-th training sample; the feature map includes at least one anchor point box; Calculate the intersection-over-union ratio between the anchor box and the true value box; Determine the first k anchor boxes with the highest intersection-over-union ratio with the true value box as positive samples corresponding to the true value box, where k is a positive integer; Determine the category information and position information corresponding to the truth frame as the category information and position information of the positive sample corresponding to the truth frame; Determine the total loss value of the i-th training sample according to the positive sample; When the total loss value does not meet the training stop condition, the model parameters of the target detection network are adjusted, and the adjusted target detection network is trained using the i+1th training sample in the training sample set until the total loss value of the target detection network meets the training stop condition, thereby obtaining the trained target detection network, where i is a positive integer.

3. The method according to claim 2, characterized in that The target detection network includes a detection head, and determining the total loss value of the i-th training sample according to the positive sample includes: Outputting the classification confidence of each anchor box in the positive sample through the classification branch of the detection head; Outputting, through the regression branch of the detection head, the offset of each anchor box in the positive sample to the true value box corresponding to the positive sample; Calculating the prediction quality of each anchor box in the positive sample according to the classification confidence and the offset; A total loss value of the i-th training sample is determined according to the prediction quality.

4. The method according to claim 3, characterized in that The determining the total loss value of the i-th training sample according to the prediction quality includes: When the prediction quality of the anchor box in the positive sample is greater than 0, the total loss value of the i-th training sample is calculated according to the following formula: Among them, L total is the total loss value, L box is the regression loss, w1 is the regression loss weight coefficient, L cls is the classification loss, w2 is the classification loss weight coefficient, L pos is the positive sample loss, w3 is the positive sample loss weight coefficient, q m is the prediction quality of the mth anchor box in the positive sample, where m is a positive integer.

5. The method according to claim 3, characterized in that: The determining the total loss value of the i-th training sample according to the prediction quality further includes: When the prediction quality of the anchor boxes in the positive samples is all 0, the loss value of the i-th training sample is calculated according to the following formula: L total =w1L box +w2L cls Among them, L total is the total loss value, L box is the regression loss, w1 is the regression loss weight coefficient, L cls is the classification loss, and w2 is the classification loss weight coefficient.

6. The method according to claim 2, characterized in that Before using the training sample set to train the target detection network to be trained until a training stop condition is met to obtain the trained target detection network, the method further includes: Acquire a pre-training data set, where the pre-training data set is a 2D image data set; The pre-training data set is used to train the initial object detection network to obtain the object detection network to be trained.

7. A point cloud target detection device, characterized in that: The device comprises: An acquisition module is used to acquire point cloud data collected by a laser radar sensor; the point cloud data includes a plurality of points, and an x-axis coordinate value, a y-axis coordinate value, a z-axis coordinate value, and a reflection intensity of each point; A point cloud column determination module, used to determine a plurality of point cloud columns corresponding to the area to be detected according to the point cloud data; each of the point cloud columns includes at least one point among the plurality of points; a pseudo image determination module, configured to determine, for each of the point cloud columns, a target pseudo image of the area to be detected according to the point cloud density of the point cloud column, the maximum, average and minimum values ​​of the x-axis coordinate values, y-axis coordinate values ​​and z-axis coordinate values ​​of all the points in the point cloud column, and the average and maximum values ​​of the reflection intensity of all the points in the point cloud column; The detection module is used to input the target pseudo image into a trained target detection network to obtain the target detection result of the area to be detected.

8. A point cloud target detection device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the method for point cloud target detection as described in any one of claims 1-6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by the processor, the method for point cloud target detection according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the method for point cloud target detection as described in any one of claims 1 to 6.

Images