Three-dimensional target detection method and device based on range view

Abstract

The present invention provides a method and apparatus for 3D object detection based on range views. The method comprises: projecting 3D point cloud data collected by a laser radar (LiDAR) into a 2D range view format using a spherical coordinate system; processing the range view using a well-established 2D object detection network, YOLOX; generating a 2D bounding box on the range view based on the scale and position information of the real 3D bounding box; creating a dataset and training the network to obtain pre-trained weights for 3D object detection; modifying the prediction branch, label assignment loss, and loss function of the 2D object detection network; and supervising the network to regress the 3D bounding box. Finally, using the range view projected into a 2D format, efficient 3D object detection using LiDAR is achieved.

Description

Technical Field

[0001] The present invention relates to a laser radar-based target detection technology, and in particular to a method and device for three-dimensional target detection by converting a point cloud into a range view. Background Art

[0002] Three-dimensional object detection is a key branch of computer vision, with widespread applications in autonomous driving, robotic environmental perception, virtual reality, and other fields. With the continuous industrialization and decreasing cost of lidar (LiDAR), 3D object detection based on lidar point cloud data has shown tremendous potential. Point clouds, consisting of clusters of points with precise 3D coordinates, offer inherent dimensionality advantages that make them incomparable to image data in 3D object detection. Furthermore, compared to cameras, which are sensitive to lighting conditions, lidar can maintain high reliability in complex environments such as darkness, rain, and fog. However, point cloud-based 3D object detection still faces numerous challenges. For example, the sheer volume of point clouds makes the computational overhead of direct neural network processing prohibitive. The sparsity and unstructured nature of point clouds also require carefully designed neural network modules, increasing the complexity of the algorithm. Therefore, optimizing the point cloud data structure, such as converting it into a range view, can improve the efficiency of 3D object detection and enhance the algorithm's practicality in real-world scenarios.

[0003] To achieve these goals, several methods have been studied for using range views for 3D vision tasks. Among them, Chen Jian et al. proposed a method for using range views for point cloud encoding and decoding (Chen Jian, Lin Yufang, Zheng Mingkui, et al. A point cloud encoding and decoding method based on vehicle-mounted lidar [P]. Chinese Patent: CN114332259A, April 12, 2022). Liu Fuchun et al. proposed a 3D semantic segmentation method based on range views (Liu Fuchun, Chen Xujian. A method and system for semantic segmentation of laser point clouds using multi-scale contextual features [P]. Chinese Patent: CN116109836A, May 12, 2023). Although these studies use range views for 3D vision tasks, the models only use simple convolutional modules or residual structures to process the range views, which limits their performance. However, given the similarities between range views and image data, we believe that some established 2D object detection networks can be well-suited for range views. Applying established 2D detection networks to range views can also, to some extent, break down the dimensional barriers between point clouds and images, facilitating further exploration of the complementary fusion of point cloud and image data. Furthermore, all of the aforementioned studies have applied range views to 3D semantic segmentation tasks. We believe that range views hold significant potential for further application in 3D object detection. Summary of the Invention

[0004] The present invention proposes a three-dimensional target detection method and device that migrates a two-dimensional target detection neural network to a range view. This method overcomes the pain points of traditional three-dimensional target detection neural networks, which have complex structures and are difficult to apply in actual scenarios, and expands the application of range views in three-dimensional target detection tasks.

[0005] The present invention's 3D target detection method based on range view first projects the coordinates of the 3D point cloud data collected by the lidar from a Cartesian coordinate system to a spherical coordinate system, and arranges the point cloud data into a range view using the tilt and pitch angles obtained by the projection. The 3D bounding box in the label file is then projected onto the 2D bounding box on the range view to train YOLOX to obtain pre-trained weights with 2D priors. YOLOX is then slightly modified to achieve the purpose of predicting the 3D target box, and a 3D loss function is added to train 3D target detection, thereby completing the 3D target detection task using a mature and streamlined YOLOX network. The specific steps are as follows:

[0006] A three-dimensional target detection method based on range view includes the following steps:

[0007] Step 1: Project the 3D point cloud data collected by the LiDAR from the Cartesian coordinate system to the spherical coordinate system to obtain a distance view;

[0008] Step 2: Project the 3D bounding box onto the 2D bounding box on the distance view and train to obtain pre-trained weights with 2D priors.

[0009] Step 3: Improve the YOLOX network model by loading pre-trained weights and performing 3D object box detection on the range view.

[0010] Step 4: Improve YOLOX's loss function and add a 3D loss function to train 3D object detection.

[0011] In step 1, the tilt angle and pitch angle in the spherical coordinate system obtained by projection are used to arrange the point cloud data into a distance view, which specifically includes the following steps:

[0012] Step 11: Convert each point in the point cloud data from its three-dimensional space x, y, z axis coordinates (x, y, z) to the coordinate origin shift distance, horizontal polar angle, vertical polar angle in the spherical coordinate system.

[0013] Step 12: To structure each point, initialize a range view matrix with a height of w and a width of h, where w is determined by the vertical resolution of the lidar and h is determined by the horizontal resolution of the lidar. Sort the point cloud from left to right in ascending azimuth angle and from top to bottom in ascending tilt angle, and fill it into the range view matrix to complete the conversion from point cloud to range view.

[0014] In step 2, the distance view is used to perform pre-training weight training, which specifically includes the following steps:

[0015] Step 21: Project the 3D bounding box in the dataset label data onto the range view to obtain a 2D bounding box. The 7-dimensional information describing the 3D bounding box in the label (x, y, z axis coordinates, length, width, height, and rotation angle (x, y, z, l, h, w, ρ)) is used to generate the 3D coordinates of the eight corner points of the 3D bounding box in the LiDAR coordinate system.

[0016] Step 22: Using the projection method in step 1, obtain the eight two-dimensional coordinates corresponding to the eight corner points of the three-dimensional bounding box on the distance view, and take the minimum and maximum values ​​in the x and y directions of the distance view to obtain (x min ,y min ) and (x max ,y max ) as the top left and bottom right corners of the distance view bounding box.

[0017] Step 23: Convert the 3D point cloud data of the KITTI dataset into a 2D distance view dataset, and convert the 3D box coordinates and category annotations into 2D VOC format 2D coordinate boxes and category annotations respectively.

[0018] Step 24: Perform YOLOX network 2D object detection training on the range view dataset to provide appropriate pre-training weights for subsequent 3D object detection tasks.

[0019] Among them, step 3 uses pre-trained weights to perform 3D object detection training, which specifically includes the following steps:

[0020] Step 31: Load the pre-trained weights into the 3D object detection network, inject 2D prior knowledge into the network, accelerate network convergence, and improve network performance.

[0021] Step 32: Modify the YOLOX network structure, and change the regression parameters in the network from two-dimensional x, y coordinates, height and width w, h four parameters to x, y, z coordinates, length, width and height l, h, w, and rotation angle ρ seven parameters, and reuse the weights of the extra prediction branch networks.

[0022] In step 4, during the label matching process of network optimization, the 3D IoU and 2D IoU between the predicted box and the target box are calculated simultaneously, and the mean of the 3D IoU and 2D IoU is used as the cost function in label assignment.

[0023] In the middle, step 4 designs a three-dimensional loss function and a non-maximum suppression method under the BEV perspective for the YOLOX network, so that the two-dimensional target detection network YOLOX can perform three-dimensional target detection in the distance view.

[0024] The second aspect of the present invention relates to a three-dimensional target detection device based on a range view, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, the device is used to implement a three-dimensional target detection method based on a range view of the present invention.

[0025] A third aspect of the present invention relates to a computer-readable storage medium, characterized in that a program is stored thereon, and when the program is executed by a processor, a three-dimensional target detection method based on a range view of the present invention is implemented.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1. This paper expands the application of range view in the field of 3D target detection and analyzes and verifies the feasibility of range view in target detection.

[0028] 2. The present invention proposes to use the weight file obtained by two-dimensional target detection through distance view as the pre-weight for three-dimensional target detection, and accelerate network convergence and improve network performance through two-dimensional prior knowledge.

[0029] 3. The present invention utilizes an advanced two-dimensional target detection network to perform three-dimensional target detection tasks on the range view, effectively overcoming the shortcomings of traditional three-dimensional target detection networks that are complex in structure and difficult to apply in actual scenarios. At the same time, the plug-and-play network structure makes this method independent of a specific two-dimensional target detection network and has strong scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a distance view projection principle diagram of the present invention.

[0031] Figure 2 is a distance view visualization diagram of the present invention.

[0032] Figure 3 The distance viewing of the present invention Figure 2 Schematic diagram of a 3D object detection dataset.

[0033] Figure 4 It is a prediction result diagram of the two-dimensional target detection network of the present invention.

[0034] Figure 5 It is the overall flow chart of the present invention.

[0035] Figure 6 This is a three-dimensional target detection prediction result diagram of the present invention. DETAILED DESCRIPTION

[0036] The technical solution of the present invention is further described below with reference to the accompanying drawings.

[0037] Example 1

[0038] The present invention proposes a three-dimensional target detection method based on range view, the flow chart is as follows Figure 5 As shown. First, the coordinates of the 3D point cloud data collected by the lidar are projected from the Cartesian coordinate system to the spherical coordinate system, and the point cloud data is arranged into a distance view using the tilt and pitch angles obtained by the projection. The 3D bounding box in the label file is then projected onto the 2D bounding box on the distance view to train YOLOX to obtain pre-trained weights with 2D priors. YOLOX is then slightly modified to achieve the purpose of predicting 3D target frames, and a 3D loss function is added to train 3D target detection, thus completing the 3D target detection task through the mature and streamlined YOLOX network. The specific steps are as follows:

[0039] Step 1: Project the 3D point cloud data collected by the LiDAR from the Cartesian coordinate system to the spherical coordinate system to obtain a distance view;

[0040] Step 2: Project the 3D bounding box onto the 2D bounding box on the distance view and train to obtain pre-trained weights with 2D priors.

[0041] Step 3: Improve the YOLOX network model by loading pre-trained weights and performing 3D object box detection on the range view.

[0042] Step 4: Improve YOLOX's loss function and add a 3D loss function to train 3D object detection.

[0043] The following will provide a detailed description of steps 1 to 4.

[0044] In step 1, the point cloud data is converted into the distance view format. The distance view is a two-dimensional data format obtained by projecting the point cloud into a spherical coordinate system. Figure 1 Compared to unordered, sparse, and unstructured point cloud data, the range view is dense and structured, and spatial information is not lost during projection. This data characteristic makes it possible to apply mature 2D object detection algorithms to the range view. The improved 2D model for the range view avoids the complexity and high computational complexity of traditional 3D models, enabling more convenient and efficient real-time object detection on edge devices. The specific steps include:

[0045] Step 11: The present invention performs 3D target detection on the distance view obtained by projecting the laser radar point cloud. Each point in the point cloud data is composed of its 3D space coordinates (x, y, z) on the x, y, and z axes. In order to obtain the distance view, it is necessary to convert the point cloud coordinates in the Cartesian coordinate system into the coordinate origin shift distance, horizontal polar angle, and vertical polar angle in the spherical coordinate system. The conversion formula is shown in formula (1).

[0046]

[0047] Step 12: After the transformation, each point in the point cloud is represented by its corresponding spherical coordinate system. To structure each point, a distance view matrix with a height of 64 and a width of 1024 is initialized. The 64 represents the number of lines of the Velodyne Lidar HDL-64E lidar in the KITTI dataset used in the experiment, that is, the number of lidar stacks at a height. The points obtained by a rotating laser beam scanning in a rotating lidar constitute a row of the matrix. The matrix width of 1024 is determined by the lidar's horizontal resolution. The HDL-64E has a horizontal resolution of approximately 0.3°, meaning that a row contains approximately 1080 points. Since the input size for target detection is generally a power of 2, the matrix width is set to 1024. Next, the azimuth angle of each point is normalized from [-π, π] to [0, 2π] and mapped to [0, 1024]. According to the scanning range of the laser radar, the pitch angle of each point is normalized from [-5°, 25°] to [0, 30°] and mapped to [0, 64]. From left to right, the azimuth angle is small to large, and from top to bottom, the point cloud is sorted and filled into the distance view matrix, thus completing the conversion from point cloud to distance view. The distance view after conversion is as follows: Figure 2 shown.

[0048] Step 2: Use the distance view to perform pre-trained weight training. In object detection network training, appropriate pre-trained weights can effectively accelerate the network training process and improve network performance. In order to verify the feasibility of object detection on the distance view, 2D object detection training on the 2D distance view can greatly assist in subsequent 3D object detection training. Therefore, the 3D bounding box in the KITTI dataset label data is projected onto the distance view to obtain a 2D bounding box. The specific steps include:

[0049] Step 21: Generate the three-dimensional coordinates of the eight corner points of the three-dimensional bounding box using the seven-dimensional information describing the three-dimensional bounding box in the label: coordinates on the x, y, and z axes, length, width, and height, and rotation angles (x, y, z, l, h, w, ρ).

[0050]

[0051] Where c0 represents the coordinates of the eight corner points, considering only the length, width, and height, without considering the rotation angle or the coordinates of the center point of the bounding box. r is the rotation matrix around the y-axis. c1 is the coordinate of the center point of the bottom surface of the bounding box. c is the 3D coordinates of the eight corner points of the resulting 3D bounding box in the camera coordinate system. The coordinates are converted from the camera coordinate system to the lidar coordinate system using the calibration matrix.

[0052]

[0053] in, is the external parameter conversion matrix from the lidar coordinate system to the camera coordinate system, R0 is the correction rotation matrix of the camera, and finally c is the three-dimensional coordinates of the eight corner points of the three-dimensional bounding box in the lidar coordinate system.

[0054] Step 22: Through the projection process of the point cloud coordinates to the distance view in step 1, the eight two-dimensional coordinates corresponding to the eight corner points of the three-dimensional bounding box on the distance view can be obtained. The minimum and maximum values ​​in the x and y directions of the distance view are taken to obtain (x min ,y min ) and (x max ,y max ) as the upper left corner and lower right corner of the 2D box on the distance view. Figure 3 shown

[0055] Step 23: After converting each frame of point cloud in the dataset as above, a two-dimensional target detection dataset of distance view is obtained, which mainly includes converting the three-dimensional point cloud data of the KITTI dataset into a two-dimensional distance view dataset, and converting the three-dimensional box coordinates and category annotations into two-dimensional VOC format two-dimensional coordinate boxes and category annotations.

[0056] Step 24: Perform 2D target detection training on the YOLOX network on the distance view dataset. After 300 training cycles, it was tested to achieve relatively accurate detection results, providing suitable pre-training weights for subsequent 3D target detection tasks. In addition, this step verifies the feasibility of using distance views for target detection. Even though a lot of semantic information is missing compared to the camera, it basically has no effect on the detection effect. It can be understood that the pre-training weights of the image dataset initially loaded have helped the neural network to better learn the semantic information on the distance view to a certain extent. The results of 2D target detection on the distance view are shown below. Figure 4 shown.

[0057] Step 3: Use pre-trained weights for 3D object detection training. In order to perform 3D object detection through YOLOX, some modifications need to be made to the network structure and training settings of YOLOX. The specific steps include:

[0058] Step 31: Load the pre-trained weights into the 3D object detection network, inject 2D prior knowledge into the network, accelerate network convergence, and improve network performance.

[0059] Step 32: During the three-dimensional target detection network training, in order to enable the network to learn the three-dimensional information of the target box, the regression parameters in the network need to be changed from the four two-dimensional parameters of x, y coordinates, height and width w, h to seven parameters of x, y, z coordinates, length, width and height l, h, w, and rotation angle ρ, and the extra prediction branch networks are weighted and reused.

[0060] In the label matching process of network optimization in step 4, in order to overcome the problem of difficulty in matching the predicted box and the target box in the early stage of training, the seven-dimensional information of the predicted box is projected onto the distance view. At the same time, the 3D intersection-over-union ratio between the predicted box and the target box and the 2D intersection-over-union ratio between the predicted box and the target box projected onto the distance view are calculated. The average of the 3D intersection-over-union ratio and the 2D intersection-over-union ratio is used as the cost function in label assignment.

[0061] The loss function in the network training in step 4 is modified to the three-dimensional box loss function. In post-processing, in order to improve the inference efficiency, the two-dimensional box of the three-dimensional prediction box under the bird's-eye view is taken for non-maximum suppression.

[0062] The results of 3D target detection based on range view are as follows Figure 6 shown.

[0063] In summary, the present invention reduces the complexity of the three-dimensional target detection network and improves the real-time performance of the detection by converting point clouds into distance views for three-dimensional target detection; at the same time, it loads the pre-trained weights obtained based on the distance view, accelerates the network convergence, and improves the network performance.

[0064] Example 2

[0065] This embodiment provides a three-dimensional object detection device based on a range view, comprising a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they implement the three-dimensional object detection method based on a range view described in Example 1.

[0066] Example 3

[0067] This embodiment relates to a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a three-dimensional target detection method based on a range view of the present invention is implemented.

[0068] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The scope of protection of the present invention should not be regarded as limited to the specific forms described in the embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A three-dimensional target detection method based on range view, characterized in that: The main steps include: Step 1: Project the 3D point cloud data collected by the LiDAR from the Cartesian coordinate system to the spherical coordinate system to obtain a distance view; Step 2: Project the 3D bounding box onto the distance view to obtain a 2D bounding box, and train to obtain pre-trained weights with 2D prior knowledge; Step 3: Improve the YOLOX network model by loading pre-trained weights and performing 3D object box detection on the range view; Step 4: Improve the loss function of YOLOX and add a 3D loss function to train 3D object detection.

2. The method for three-dimensional target detection based on range view according to claim 1, characterized in that: In step 1, the tilt and pitch angles in the projected spherical coordinate system are used to arrange the point cloud data into a distance view. The specific steps include the following: Step 11: Convert each point in the point cloud data from its three-dimensional space x, y, z axis coordinates (x, y, z) to the coordinate origin shift distance, horizontal polar angle, vertical polar angle in the spherical coordinate system. Step 12: To arrange each point in a structured manner, initialize a distance view matrix with a height of w and a width of h, where w is determined by the vertical resolution of the lidar and h is determined by the horizontal resolution of the lidar. Sort the point cloud from left to right according to the azimuth angle and from top to bottom according to the tilt angle into the distance view matrix to complete the conversion from point cloud to distance view.

3. The method for three-dimensional target detection based on range view according to claim 1, characterized in that: In step 2, the distance view is used to perform pre-training weight training, which includes the following steps: Step 21: Project the 3D bounding box in the dataset label data onto the range view to obtain a 2D bounding box; generate the 3D coordinates of the eight corner points of the 3D bounding box in the lidar coordinate system using the seven-dimensional information (x, y, z axis coordinates, length, width, height, and rotation angle (x, y, z, l, h, w, ρ)) describing the 3D bounding box in the label; Step 22: Using the projection method in step 1, obtain the eight two-dimensional coordinates corresponding to the eight corner points of the three-dimensional bounding box on the distance view, and take the minimum and maximum values ​​in the x and y directions of the distance view to obtain (x min ,y min ) and (x max ,y max ) as the upper left and lower right corners of the distance view bounding box; Step 23: Convert the 3D point cloud data of the KITTI dataset into a 2D distance view dataset, and convert the 3D box coordinates and category annotations into 2D VOC format 2D coordinate boxes and category annotations respectively; Step 24: Perform YOLOX network 2D object detection training on the range view dataset to provide appropriate pre-training weights for subsequent 3D object detection tasks.

4. The method for three-dimensional target detection based on range view according to claim 1, characterized in that: In step 3, pre-trained weights are used for 3D object detection training, which specifically includes the following steps: Step 31: Load the pre-trained weights into the 3D object detection network to inject 2D prior knowledge into the network, accelerate network convergence, and improve network performance; Step 32: Modify the YOLOX network structure, and change the regression parameters in the network from two-dimensional x, y coordinates, height and width w, h four parameters to x, y, z coordinates, length, width and height l, h, w, and rotation angle ρ seven parameters, and reuse the weights of the extra prediction branch networks.

5. The method for three-dimensional target detection based on range view according to claim 1, characterized in that: In step 4, during the label matching process of network optimization, the 3D IoU and 2D IoU between the predicted box and the target box are calculated simultaneously, and the mean of the 3D IoU and 2D IoU is used as the cost function in label assignment.

6. The method for three-dimensional target detection based on range view according to claim 1, characterized in that: Step 4 designs a three-dimensional loss function and a non-maximum suppression method under the BEV perspective for the YOLOX network, so that the two-dimensional target detection network YOLOX can perform three-dimensional target detection in the distance view.

7. A three-dimensional target detection device based on range view, characterized in that: The invention comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a three-dimensional target detection method based on a range view according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, a three-dimensional target detection method based on a range view according to any one of claims 1 to 6 is implemented.

Images