A real-time 3D object detection method based on point cloud cross-view feature conversion

Abstract

The present invention discloses a real-time three-dimensional target detection method based on point cloud cross-view feature conversion. The three-dimensional point cloud acquired by laser radar is projected into a two-dimensional ring view; a neural network structure is established, and a point-surround view feature extraction module is used to extract features from two formats of the three-dimensional point cloud and the two-dimensional ring view to obtain a three-dimensional feature point cloud; the three-dimensional feature point cloud is gridded using a layered voxel column module, and finally projected onto a bird's-eye view to obtain bird's-eye view features; the neural network structure is trained to obtain a trained neural network structure, and the laser radar point cloud to be tested is input into the neural network structure, and then the three-dimensional frame predicted by the scene is calculated. The present invention effectively improves the detection accuracy of three-dimensional target detection based on laser radar point cloud, can make full use of the feature advantages between each view of the point cloud, and only uses one-dimensional and two-dimensional convolutions to complete three-dimensional target detection, thereby improving the three-dimensional target detection accuracy under high real-time conditions.

Description

Technical Field

[0001] The present invention relates to a target detection method based on laser radar in the technical field of unmanned vehicle and robot visual navigation, and in particular to a real-time three-dimensional target detection method based on point cloud cross-view feature conversion. Background Art

[0002] Object detection refers to finding objects in a perceptible environment and regressing their position and size information. It is a key technology to ensure the safe operation of complex systems such as unmanned driving and autonomous robots. Convolutional neural networks have made great progress in the field of image-based two-dimensional object detection. These deep networks use operations such as 2D convolution to extract high-level semantic information from images to understand the information of the images. Compared with traditional methods, the effect is significantly improved and it has quickly become the mainstream method in the field of object detection. However, image-based two-dimensional object detection lacks depth information and cannot be directly used for three-dimensional perception and navigation in real scenes. In order to provide accurate perception information to autonomous vehicles and robots, three-dimensional object detection came into being. In three-dimensional object detection, lidar is a common distance sensor. The point cloud collected provides accurate three-dimensional information, which provides a basis for accurate three-dimensional object detection.

[0003] Three-dimensional object detection can be divided into point-based, voxel-based and view-based methods according to the representation of the input point cloud and the means of feature extraction.

[0004] The point-based method directly uses the points in the 3D point cloud as the smallest unit for feature extraction. Through point cloud sampling, without any quantization loss, it fully extracts the 3D geometric information features of the point cloud itself, and regresses the 3D box based on the point. It is the most basic point cloud 3D object detection algorithm. For example, PointNet and PointNet++ designed a basic network of point clouds to extract global and local features. Subsequently, Point RCNN proposed an end-to-end two-stage point cloud detection model. The method first uses PointNet++ to segment the global point cloud into foreground and background to obtain all foreground point clouds, and then regresses a 3D box for the local features of each point, and finally obtains the final detection result through non-maximum suppression. However, since the feature transfer module in PointNet++ needs to calculate the distance between any two points in the point cloud, the complexity is O(N2), so most of these algorithms have poor real-time performance and cannot be well applied.

[0005] Voxel-based methods sample point clouds into regular 3D spaces, then extract features using operations such as convolution like processing 2D images, and finally regress 3D boxes from voxelized features. VoxelNet proposed a concise framework, the main idea of ​​which is to quantize point clouds into a uniform 3D voxelized grid, and then regress the 3D box from the bird's-eye view. On this basis, SECOND proposed the operation of 3D sparse convolution, which greatly accelerated the real-time performance of the network. PV RCNN proposed a two-branch skeleton network that extracts features from points and voxels at the same time, and uses the progressive fusion of voxel features and point features to obtain more refined features. These voxel-based 3D object detection algorithms benefit from voxel partitioning algorithms and 3D sparse convolutions, and can efficiently and quickly perform accurate 3D region boxes. However, the implementation and acceleration of 3D sparse convolutions are relatively troublesome, with high system requirements and limited application scenarios.

[0006] The view-based method projects the point cloud onto a certain view (such as a ring view, a bird's-eye view, etc.), and then directly uses 2D convolution to complete the regression of the 3D box like 2D target detection. Pixor projects the 3D point cloud of the lidar into a bird's-eye view and completes the detection from the bird's-eye view. PointPillars, as a continuation of the voxelization method, uses a PointNet to extract features in each voxel column, represents the point cloud as a bird's-eye view, and directly regresses the 3D box on the pseudo bird's-eye view feature map. The view-based method only uses 1D and 2D convolutions to complete the detection of 3D targets, which has a strong real-time advantage. However, since directly projecting the 3D point cloud onto a certain view will lose the information of the compressed dimension, the accuracy is relatively low. Summary of the invention

[0007] In order to solve the problems existing in the background technology, the purpose of the present invention is to realize three-dimensional target detection in the scene by fully utilizing the advantages of multiple views through feature conversion of point clouds across views, and to be suitable for an efficient perception system for real-time acquisition and real-time processing.

[0008] The present invention provides a real-time 3D target detection method based on point cloud cross-view feature conversion, which fully improves the detection accuracy of view-based methods and achieves a perfect balance between accuracy and real-time performance. First, a point-surround view feature extraction module is designed to extract and fuse features of 3D point clouds and 2D surround views at the same time. Then, a hierarchical voxel column module is proposed to gradually compress 3D feature point clouds into bird's-eye view features while retaining sufficient 3D information. Finally, a sparse detection head module is proposed to classify and regress valid grids with points in the bird's-eye view features. This method, combined with a simple and real-time network structure, reaches the best level in view-based methods and completes the task of real-time and efficient 3D target detection.

[0009] The present invention can effectively overcome the problems of complex network, low real-time performance and low efficiency in 3D object detection, and use a skeleton network composed of pure 1D and 2D convolution to extract cross-view features of point clouds. The network used is feature-rich, structure-concise, computationally inexpensive and highly real-time.

[0010] The steps of the technical solution adopted by the present invention are as follows:

[0011] 1) Collect the 3D point cloud of the scene through the laser radar, and perform surround projection on the 3D point cloud to obtain the corresponding 2D surround view;

[0012] 2) Establish a neural network structure, use the known data set and the corresponding two-dimensional ring view to train the neural network structure, set the total loss function to supervise the neural network structure, and obtain a trained neural network structure;

[0013] 3) The three-dimensional point cloud and the corresponding two-dimensional ring view of the scene to be tested are input into the trained neural network structure, and the regression residual of the neural network structure and the classification map of the current scene to be tested are output. The predicted three-dimensional box is calculated based on the classification map and regression residual of the current scene to be tested to obtain the final predicted three-dimensional box.

[0014] In the step 1), a two-dimensional annular view with the pitch angle and the heading angle as the coordinate axes is established, and the pitch angle and the heading angle of each point in the three-dimensional point cloud and the origin of the laser radar coordinate system are calculated and projected into the two-dimensional annular view with the pitch angle and the heading angle as the coordinate axes to obtain the corresponding two-dimensional annular view.

[0015] In the step 2), the neural network structure includes a point-surround view feature extraction module, a hierarchical voxel column module and a sparse detection head module; the input of the neural network structure is input into the point-surround view feature extraction module, the point-surround view feature extraction module is connected to the sparse detection head module after passing through the hierarchical voxel column module, and the output of the sparse detection head module is used as the output of the neural network structure.

[0016] The point-surround view feature extraction module includes a surround view branch, a point branch and three point-surround view feature fusion blocks, and the surround view branch is connected to the point branch through the three point-surround view feature fusion blocks; the surround view branch includes six encoding modules and four decoding modules, and the point branch includes three 1D convolutional layers;

[0017] The three-dimensional point cloud is input to the first 1D convolutional layer, and the two-dimensional surround view is input to the first encoding module. The first point cloud feature P1 output by the first 1D convolutional layer and the first surround view feature E1 output by the first encoding module are both input to the first point-surround view feature fusion block. The first point-surround view feature fusion block outputs the updated first point cloud feature and the first surround view feature respectively. The updated first point cloud feature is input to the second 1D convolutional layer. The updated first surround view feature is input to the second encoding module. The second encoding module is connected to the sixth encoding module after passing through the third encoding module, the fourth encoding module and the fifth encoding module in sequence. The second point cloud feature P2 output by the second 1D convolutional layer and the sixth surround view feature E6 output by the sixth encoding module are both input to the second In the point-surround view feature fusion block, the second point-surround view feature fusion block outputs the updated second point cloud feature and the sixth surround view feature respectively, the updated second point cloud feature is input into the third 1D convolution layer, the updated sixth surround view feature is input into the first decoding module, the first decoding module is connected to the fourth decoding module after passing through the second decoding module and the third decoding module in sequence, the third point cloud feature P3 output by the third 1D convolution layer and the tenth surround view feature D4 output by the fourth decoding module are both input into the third point-surround view feature fusion block, the third point-surround view feature fusion block outputs the updated third point cloud feature and the tenth surround view feature respectively, the updated third point cloud feature is recorded as a 3D feature point cloud and used as the output of the point-surround view feature extraction module;

[0018] In the point-surround view feature fusion block, first, according to the two-dimensional surround view corresponding to the point cloud feature, each point of the point cloud feature is sampled by a bilinear interpolation method to obtain the 2D feature of the point, and the input surround view feature is used as the 3D feature of the point. The 2D feature of the point and the 3D feature of the point are cascaded and input into three 1D convolutional layers connected in sequence. The output of the last 1D convolutional layer is used as the updated point cloud feature, and the output of the last 1D convolutional layer is annularly back-projected to obtain the updated surround view feature.

[0019] Each encoding module is composed of a dense residual block and several convolutional layers connected in sequence. The input of each encoding module is input into the current dense residual block, and the output of the last convolutional layer is used as the output of the current encoding module. The decoding module is composed of a dense residual block and several deconvolutional layers connected in sequence. The input of each decoding module is input into the current dense residual block, and the output of the last deconvolutional layer is used as the output of the current decoding module.

[0020] In the hierarchical voxel column module, the 3D feature point cloud input by the point-surround view feature extraction module is first voxelized to obtain a 3D point cloud mesh feature;

[0021] Then, the 3D point cloud mesh features are input into the 1D and 2D convolutional layers in turn to extract features in the height dimension and length and width dimension, and after channel reorganization, the bird's-eye view features are obtained and output.

[0022] The sparse detection head module includes a classification head and a regression head based on anchor frames;

[0023] In the sparse detection head module, a 3D anchor frame is first laid for each point grid in the bird's-eye view feature output by the hierarchical voxel column module. The parameters of the 3D anchor frame are determined by the position of the 3D anchor frame (x, y, z), the average size of the detection target (w mean , l mean ,h mean ) and orientation angles, each point grid has two 3D anchor frames with different orientation angles; then the 3D anchor frames are divided into positive and negative samples according to the intersection-over-union ratio between the 3D anchor frames and the true frames;

[0024] Then, based on the classification head of the anchor frame, the possibility of each point grid divided into positive and negative samples in the bird's-eye view feature being the foreground is predicted, and the predicted classification probability of each point grid is obtained. Based on the predicted classification probability of each point grid, a predicted classification probability set is obtained, and based on the predicted classification probability set, a classification map of the current scene is obtained and output;

[0025] Finally, in the anchor-based regression head, the preset residual of the three-dimensional anchor frame is first set, and the anchor frame parameters are regressed for the three-dimensional anchor frames divided into positive and negative samples in the foreground grid of the current bird's-eye view feature to obtain the predicted frame, and the regression residual between the predicted frame and the true frame is calculated and output.

[0026] The total loss function includes the 3D box classification loss function, the 3D box regression loss function and the angle classification loss function, and the calculation formula is as follows:

[0027]

[0028] Among them, L represents the total loss function value, L cls represents the 3D box classification loss function, L loc Represents the 3D box regression loss function value, L dir Represents the angle classification loss function value, n pos is the number of positive samples in all 3D anchor boxes, λ loc、 λ cls , dir are the first, second, and third loss weights respectively.

[0029] The three-dimensional box classification loss function is calculated based on the predicted classification probability set, the three-dimensional box regression loss function is calculated based on the preset residual and the regression residual, and the angle classification loss function is obtained by classifying the orientation according to the orientation angle of the predicted box output by the regression head to obtain the front and back orientation of the detected target, which is calculated based on the front and back orientation of the detected target.

[0030] The preset residual of the 3D anchor frame is calculated by 7 parameters of the real frame and 7 parameters corresponding to the 3D anchor frame. The specific formula is as follows:

[0031]

[0032]

[0033] Δθ=sin(θ gt -θ a )

[0034] Among them, Δx is the preset horizontal coordinate residual, x gt and x a 分 are the horizontal coordinates of the real frame and the 3D anchor frame, Δy is the preset vertical coordinate residual, y gt and a are the ordinates of the real frame and the 3D anchor frame respectively, Δz is the preset ordinate residual, z gt and z a are the vertical coordinates of the real box and the 3D anchor box, Δw is the preset box width residual, and w gt and w a are the frame widths of the real frame and the 3D anchor frame, Δl is the preset frame length residual, l gt and l a are the frame lengths of the real frame and the 3D anchor frame, Δh is the preset frame height residual, and h gt and h a are the heights of the real box and the 3D anchor box, Δθ is the preset angle residual, and θ gt and θ a are the orientation angles of the real box and the 3D anchor box respectively, and d represents the diagonal length of the 3D anchor box, satisfying

[0035] In the step 3), all grids whose target confidence is greater than the target discrimination threshold in the classification map of the current scene to be tested are extracted and used as the location of the detected target, and the parameters of the predicted three-dimensional box of the location of the detected target are calculated according to the regression residual of the neural network structure. The calculation formula is as follows:

[0036] x=x a +d a Δx′, y=y a +d a Δy ′, z=z a +d a Δz′

[0037] w=w a e Δw′ , l=l a e Δl′ , h=h a e Δh′

[0038] θ=arcsin(θ a +Δθ′)

[0039] Among them, Δx′, Δy′, Δz′, Δw′, Δl′, Δh′, and Δθ′ are respectively the regression horizontal coordinate residual, regression vertical coordinate residual, regression vertical coordinate residual, regression frame width residual, regression frame length residual, regression frame height residual, and regression angle residual; x represents the horizontal coordinate of the predicted three-dimensional frame, and x represents the horizontal coordinate of the predicted three-dimensional frame. a represents the horizontal coordinate of the 3D anchor box, y represents the vertical coordinate of the predicted 3D box, a represents the ordinate of the 3D anchor box, z represents the vertical coordinate of the predicted 3D box, a represents the vertical coordinate of the 3D anchor box, w represents the box width of the predicted 3D box, and w a represents the width of the 3D anchor box, l represents the length of the predicted 3D box, l a represents the length of the 3D anchor box, h represents the height of the predicted 3D box, and h a represents the height of the 3D anchor box, θ represents the orientation angle of the predicted 3D box, and θ a represents the orientation angle of the 3D anchor box, d represents the diagonal length of the 3D anchor box, satisfying

[0040] The present invention first projects the input lidar point cloud to obtain a two-dimensional surround view form, and uses a point-surround view feature extraction module to extract features from the three-dimensional point cloud form and the two-dimensional surround view form and fuse them; then uses a layered voxel column module to compress the point cloud features into features on a bird's-eye view; finally, uses a sparse detection head module to classify and regress the three-dimensional box of the valid grid with existing points, and outputs the position and posture of the three-dimensional box of the current scene.

[0041] This paper explores the advantages of different point cloud views, extracts features from a dense two-dimensional ring view, uses point cloud views for feature transfer and compression, and finally performs detection under an unobstructed and scale-consistent bird's-eye view. The representation of features is constantly transformed with the point cloud view, making full use of the advantages of different point cloud views at different stages of the network, and constructing a framework for cross-view feature extraction and conversion.

[0042] The skeleton network of the present invention is composed of 1D and 2D convolutions, and the complex 3D convolution is abandoned. In order to learn the 3D information of the point cloud to obtain 3D features, the layered voxel column method is used to retain sufficient 3D information during the point cloud feature conversion. At the same time, the overall network is accelerated in the feature extraction and sparse detection of the network. The whole method achieves a balance and unity between accuracy and real-time performance, and has high application value.

[0043] The present invention constructs a real-time 3D target detection method based on point cloud cross-view feature conversion. The laser radar point cloud is input, and the 3D point cloud is projected to obtain a 2D surround view. The point-surround view feature extraction module is used to extract features from the 3D point cloud and the 2D surround view at the same time, and the point-surround view feature fusion block is used to fuse the two-way features in the early, middle and late stages of the network, and output a 3D feature point cloud; then the layered voxel column module is used to compress the feature dimension of the 3D feature point cloud to obtain the bird's-eye view feature; finally, the sparse detection head module is used to classify and regress the grid with points, and the 3D box position and posture in the scene are output. For the learning of parameters in the network structure, a total loss function is set to supervise the training of the classification head and the regression head.

[0044] Compared with the background technology, the present invention has the following beneficial effects:

[0045] (1) The present invention can effectively utilize the advantages of point clouds in different views, perform feature conversion from surround view to point cloud view to bird's-eye view, and design a framework for cross-view feature extraction and conversion, which fully improves the accuracy of view-based 3D object detection.

[0046] (2) The present invention designs a hierarchical voxelization module, which uses 2D convolution to gradually convert 3D point cloud features into bird's-eye view features. During the conversion process, features are first extracted in the height dimension, and then in the length and width dimensions. This progressive feature extraction method can better preserve the 3D position information of the point cloud and improve the accuracy of detection under the bird's-eye view.

[0047] (3) The present invention designs a sparse detection head that only classifies and regresses valid grids with points, which can greatly reduce the burden on the detection head. At the same time, it can filter out most of the pointless areas as negative samples and balance the positive and negative sample ratio of the classification head.

[0048] (4) The present invention completes three-dimensional target detection by simply combining 1D convolution and 2D convolution, discarding the heavy 3D convolution, reducing the capacity of the network model, improving the efficiency of the method, and achieving a leading real-time level.

[0049] In general, the network structure of the present invention has strong real-time performance and high precision. All modules of the network can be accommodated in an end-to-end volume and network, and the structure is briefly introduced; the point-surround view feature extraction module can effectively extract the three-dimensional position information and two-dimensional texture information of the point cloud, and fully mine the 3D information of the point cloud; the hierarchical voxelization module uses 1D and 2D convolutions to complete the conversion of the point cloud view to the bird's-eye view, retaining the 3D information of the point cloud; the sparse detection head module only classifies and regresses the effective grids in the point area, reducing the network burden and improving the real-time performance of the network. This solution is suitable for various outdoor scenes, has strong robustness, and has reached a leading level in accuracy and real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of the method of the present invention;

[0051] Figure 2 is the neural network structure of the method of the present invention;

[0052] Figure 3 It is a structural diagram of the point-surround view feature fusion block of the method of the present invention;

[0053] Figure 4 is a hierarchical voxel column module of the method of the present invention;

[0054] Figure 5 It is the visualization output of the method of the present invention on the validation set of the KITTI 3D Object Detection dataset;

[0055] Figure 6 This is an example of a bird's-eye view of the point cloud from the KITTI 3D Object Detection dataset ID=000798. DETAILED DESCRIPTION

[0056] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0057] like Figure 1 As shown in the flowchart of , the embodiment and implementation process of the complete method of the present invention are as follows:

[0058] Taking the KITTI 3D Object Detection dataset as a known dataset and three-dimensional object detection as an example, the idea and specific implementation steps of the real-time three-dimensional object detection method based on point cloud cross-view feature conversion are explained.

[0059] The lidar point cloud of the embodiment and the true value 3D box used for supervision are both from the KITTI 3D Object Detection known dataset.

[0060] Using the division of the known KITTI 3D Object Detection dataset, a total of 7481 data containing true 3D boxes are provided, including 3712 training sets and 3769 validation sets. The training set and validation set belong to different scenes and sequences and have no intersection with each other. Both the training set and validation set have true 3D boxes. For the lidar point cloud provided by the training set, perform steps 1 to 2;

[0061] The present invention comprises the following steps:

[0062] 1) Collect the 3D point cloud of the scene through the laser radar, and perform surround projection on the 3D point cloud to obtain the corresponding 2D surround view;

[0063] In step 1), a two-dimensional ring view with pitch angle and heading angle as coordinate axes is established, and the pitch angle and heading angle of each point in the three-dimensional point cloud and the origin of the laser radar coordinate system are calculated, and projected into the two-dimensional ring view with pitch angle and heading angle as pixel coordinate axes to obtain the corresponding two-dimensional ring view. Among them, the number of channels for each point in the three-dimensional point cloud is 5, including the original coordinates of the point cloud (x, y, z), the viewing distance r and the reflection intensity i.

[0064] The calculation formulas for the pitch angle and heading angle are as follows:

[0065]

[0066] Where (x, y, z) is the coordinate of the laser radar point cloud, (u, v) is the pixel coordinate of the point on the two-dimensional ring view, h and w are the height and width of the two-dimensional ring view. In the KITTI dataset, the default is a 64-line mechanical laser radar, so there are 64 channels in the pitch angle, that is, in the vertical direction, and 2048 channels are used to represent the 360° heading angle range, that is, h = 64, w = 2048. fov and fov up is the pitch angle of the mechanical laser radar and the upper pitch angle above the horizontal line. In the KITTI dataset, it is set to fov = 28°. up = 3°. r is the distance of each point from the center of the LiDAR. When multiple points are projected onto the same pixel coordinate, the closer point is retained.

[0067] At the same time, since the KITTI dataset only annotates the 3D box within the 90° range of the front view, the 2D ring view is cropped to retain only the contents of 512 channels in the range of the front view, and finally a 64×512×5 2D ring view is generated.

[0068] 2) Establish a neural network structure, use the N×4 three-dimensional point cloud and the corresponding 64×512×5 two-dimensional ring view in the known data set to train the neural network structure, set the total loss function to supervise the neural network structure, and obtain the trained neural network structure;

[0069] In step 2), the neural network structure includes a point-surround view feature extraction module, a hierarchical voxel column module and a sparse detection head module; the input of the neural network structure is input into the point-surround view feature extraction module, the point-surround view feature extraction module is connected to the sparse detection head module after passing through the hierarchical voxel column module, and the output of the sparse detection head module is used as the output of the neural network structure.

[0070] like Figure 2 As shown in the figure, the point-surround view feature extraction module includes a surround view branch, a point branch and three point-surround view feature fusion blocks. The surround view branch is connected to the point branch through three point-surround view feature fusion blocks; the surround view branch extracts features of the two-dimensional surround view through a 2D convolutional basic network composed of dense residual blocks; the point branch extracts features through a simple 1D convolutional network; at the same time, three point-surround view feature fusion blocks are set up between the two branches to perform feature interaction in the early, middle and late stages of the network, and finally output a 3D feature point cloud.

[0071] The ring view branch includes six encoding modules and four decoding modules. The six encoding modules constitute the encoder of the ring view branch, and the four decoding modules constitute the decoder of the ring view branch. The point branch includes 3 1D convolutional layers;

[0072] The three-dimensional point cloud is input to the first 1D convolutional layer, and the two-dimensional surround view is input to the first encoding module. The first point cloud feature P1 output by the first 1D convolutional layer and the first surround view feature E1 output by the first encoding module are both input to the first point-surround view feature fusion block. The first point-surround view feature fusion block outputs the updated first point cloud feature and the first surround view feature respectively. The updated first point cloud feature is input to the second 1D convolutional layer. The updated first surround view feature is input to the second encoding module. The second encoding module is connected to the sixth encoding module after passing through the third encoding module, the fourth encoding module and the fifth encoding module in sequence. The second point cloud feature P2 output by the second 1D convolutional layer and the sixth surround view feature E6 output by the sixth encoding module are both input to the second In the point-surround view feature fusion block, the second point-surround view feature fusion block outputs the updated second point cloud feature and the sixth surround view feature respectively, the updated second point cloud feature is input into the third 1D convolutional layer, the updated sixth surround view feature is input into the first decoding module, the first decoding module is connected to the fourth decoding module after passing through the second decoding module and the third decoding module in sequence, the third point cloud feature P3 output by the third 1D convolutional layer and the tenth surround view feature D4 output by the fourth decoding module are both input into the third point-surround view feature fusion block, the third point-surround view feature fusion block outputs the updated third point cloud feature and the tenth surround view feature respectively, the updated third point cloud feature is recorded as a 3D feature point cloud and used as the output of the point-surround view feature extraction module.

[0073] Each encoding module is composed of a dense residual block and several convolutional layers connected in sequence. The input of each encoding module is input into the current dense residual block, and the output of the last convolutional layer is used as the output of the current encoding module. The decoding module is composed of a dense residual block and several deconvolutional layers connected in sequence. The input of each decoding module is input into the current dense residual block, and the output of the last deconvolutional layer is used as the output of the current decoding module.

[0074] like Figure 3 As shown in the figure, in the point-surround view feature fusion block, firstly, according to the 2D surround view corresponding to the point cloud feature, the bilinear interpolation method is used to sample each point of the point cloud feature to obtain the 2D feature of the point, and the input surround view feature is used as the 3D feature of the point. The 2D feature of the point and the 3D feature of the point are cascaded and input into three 1D convolutional layers connected in sequence. The output of the last 1D convolutional layer is used as the updated point cloud feature, and the output of the last 1D convolutional layer is annularly back-projected to obtain the updated surround view feature. The annular back-projection is the inverse process of the annular projection in step 1).

[0075] The encoder first extracts low-dimensional features from the input two-dimensional ring view through two dense residual blocks with a step size of 1 and several convolutions to obtain the first ring view feature E1 and the second ring view feature E2, and then gradually downsamples the image to 2 times E3, 4 times E4, 8 times E5, and 16 times E6 through dense residual blocks with a step size of 2 and several convolutions. Subsequently, the decoder uses a dense residual block with a step size of 1 and bilinear interpolation to upsample and gradually restore it to the original size. The features are represented as 8 times D1, 4 times D2, 2 times D3, and 1 times D4, respectively.

[0076] The point branch uses a simple 1D convolutional network to extract the 3D features of the point cloud position. In order to better fuse the features between the two, three point-surround view feature fusion blocks are designed to interact with features in the early, middle and late stages of the network, which occur on the original size feature E1 of the surround view, the 16x downsampled feature E6 and the original size feature D4 of the decoder.

[0077] The specific processing process of the point-surround view feature extraction module is as follows:

[0078] Ring view branch: Input a five-channel two-dimensional ring view of size H×W×5. The ring view branch is an encoder-decoder structure. The encoder part includes 6 dense residual blocks with step sizes of (1, 1, 2, 2, 2, 2). After each dense residual block, several ordinary convolutions are added to fully extract the features at that size. The number is {3, 3, 5, 5, 5, 5}. The first two dense residual blocks and the corresponding convolutions are mainly used to fully extract the low-dimensional features E1 and E2 at the original size. The last four dense residual blocks and the corresponding convolutions are mainly used to gradually downsample. The decoder consists of 4 dense residual blocks with step sizes of (1, 1, 1, 1). Each dense residual block has an upsampling operation based on bilinear interpolation, which extracts features to H×W feature map.

[0079] Point branch: Input is a 4-channel N×4 3D point cloud. The point branch consists of 3 basic networks of 1D convolution, which are used 3 times in total to obtain P1, P2, and P3 features with different numbers of channels respectively.

[0080] Table 1: Network structure table of ring view branches and point branches

[0081]

[0082] Point-surround view feature fusion block: Input a surround view feature in the surround view branch and a point cloud feature in the point branch, taking E1 and P1 as examples. First, for each point in the point cloud feature P1, according to its pixel position on the surround view, the corresponding 2D feature PE1 of the point is sampled by bilinear interpolation. Then E1 and PE1 are cascaded, and the updated features are obtained through 3 1D convolutions. Finally, the updated features are input into the next module, and then projected and input into the next module. The updated features continue to be extracted in their respective branches. The point-surround view feature fusion block is performed between E1 and P1, E6 and P2, and D4 and P3, and is used 3 times in total.

[0083] Output: After the above-mentioned surround view branch and point branch, as well as the point-surround view feature fusion blocks interspersed therein, the point-surround view feature extraction module finally outputs the N×64 updated third point cloud features.

[0084] like Figure 4 As shown, in the hierarchical voxel column module, the input 3D feature point cloud is first voxelized through 1D and 2D convolutions to obtain a three-dimensional point cloud mesh feature H×W×D×C. In the specific implementation, the voxelization resolution is set according to the range of the detection space, and the space is divided into a three-dimensional grid of H×W×D; for each point in the 3D feature point cloud, its grid coordinates are calculated according to the spatial range and resolution, and filled into the grid to obtain a three-dimensional point cloud mesh feature H×W×D×C.

[0085] Then, the three-dimensional point cloud mesh features H×W×D×C are input into the 1D and 2D convolution layers in turn to extract features in the height dimension and length and width dimension. The 3D features are gradually compressed into bird's-eye view features through 1D and 2D convolutions. Then, after channel reorganization, the bird's-eye view features are obtained and output.

[0086] High-dimensional feature extraction: High-dimensional feature extraction consists of 1D convolution. For the input 3D point cloud mesh feature H×W×D×C, firstly, a rearrangement operation is performed to integrate the last two dimensions together to obtain the feature of H×W×(D×C), and then a 1D convolution is used on the last dimension, with the number of input channels being (D×C) and the number of output channels being C, to obtain the first bird's-eye view feature of H×W×C.

[0087] Length and width dimension feature extraction: The first bird's-eye view feature is input into the 2D convolution layer. The length and width dimension feature extraction consists of 2D convolution. For the bird's-eye view feature of H×W×C, 3 2D convolutions with a step size of 1 are used for feature extraction. The second bird's-eye view feature of H×W×C is still output as the final bird's-eye view feature. In the specific implementation, the scene range is preset in the KITTI dataset as x∈[0, 69.12], y∈[-39.68, 39.68], z∈[-3, 1], the unit is m, and the voxel resolution of the three dimensions is preset as (0.16, 0.16, 0.2). Therefore, the feature size of the three-dimensional point cloud grid is H=496, W=432, D=32, C=64. Finally, the hierarchical voxel column module outputs a bird's-eye view feature of size 496×432×64.

[0088] The sparse detection head module screens all grids in the bird's-eye view feature, extracts all grids with points, and then uses 2D convolution to classify and regress the detection boxes in units of grids. Due to the sparsity of point clouds, most grids in the bird's-eye view do not have point clouds. Therefore, in order to save computing resources, the sparse grids with points in the bird's-eye view feature can be directly classified and regressed.

[0089] The sparse detection head module includes a classification head and a regression head based on anchor frames. In the sparse detection head module, a 3D anchor frame is first laid on each point grid in the bird's-eye view feature. The parameters of the 3D anchor frame are determined by the position of the 3D anchor frame (x, y, z), the average size of the detection target (w mean , l mean ,h mean) and the orientation angle is 0° or 90°. The number of orientation angles determines the number of 3D anchor frames for each grid, that is, in this embodiment, each foreground grid has two 3D anchor frames with different orientation angles; then the 3D anchor frames are divided into positive and negative samples according to the size of the intersection over union (IoU) between the 3D anchor frames and the true frames; in the specific implementation, the position center is the grid center, the size is the average size obtained after traversing the true value 3D frames in the KITTI dataset, for the Car class, w=1.60, l=3.90, h=1.56, the orientation angle is set to 0° and 90°, for target objects such as vehicles, the intersection over union ratio is greater than 0.6 for positive samples, and less than 0.45 for negative samples. The classification head performs the confidence classification task of the anchor frame, the regression head performs the parameter estimation task of the anchor frame relative to the true value frame, and uses the total loss function to calculate the total loss of the predicted 3D frame output by each frame of the lidar point cloud in the dataset, and trains each parameter in the neural network structure to minimize the total loss to achieve the effect of supervised learning.

[0090] Then, in the classification head based on the anchor frame, three 2D convolutional layers connected in sequence are used to predict the possibility of whether each point grid divided into positive and negative samples in the bird's-eye view features is the foreground, and the predicted classification probability of each point grid is obtained. Based on the predicted classification probability of each point grid, a predicted classification probability set is obtained, and based on the predicted classification probability set, a classification map with a three-dimensional anchor frame of the current scene is obtained and output;

[0091] Finally, the preset residual of the three-dimensional anchor frame is first set in the regression head based on the anchor frame, and the three-dimensional anchor frame divided into positive and negative samples in the foreground grid of the current bird's-eye view feature is regressed through three sequentially connected 2D convolutional layers to obtain the predicted frame, and the regression residual between the predicted frame and the real frame is calculated and output; that is, the 7 parameters of the real frame and the 7 parameters corresponding to the predicted frame are calculated, which is similar to the calculation formula of the preset residual. The foreground grid is a point grid containing the detection target in the bird's-eye view feature. In this embodiment, the detection target is a vehicle.

[0092] The total loss function includes the 3D box classification loss function, the 3D box regression loss function and the angle classification loss function. The calculation formula is as follows:

[0093]

[0094] Among them, L represents the total loss function value, L cls represents the 3D box classification loss function, L loc Represents the 3D box regression loss function value, L dir Represents the angle classification loss function value, n pos is the number of positive samples in all 3D anchor boxes, λ loc , cts ,dir are the first, second, and third loss weights respectively. In the specific implementation, the three loss weights are λ loc =2,λ cls =1,λ dir =0.2.

[0095] The 3D box classification loss function is calculated based on the predicted classification probability set. In the specific implementation, the 3D box classification loss function is the Focal Loss loss function, and the calculation formula is as follows:

[0096] L cls = -α(1-p) γ log(p)

[0097] Among them, L cls is the classification loss function value of the point grid, p is the predicted classification probability of the current three-dimensional anchor box of the point grid, α is the first hyperparameter, γ is the second hyperparameter, the first hyperparameter α=0.25, and the second hyperparameter γ=2.

[0098] The 3D box regression loss function is calculated based on the preset residual and the regression residual. In the specific implementation, the regression loss function uses the Smooth L1 Loss loss function, and the calculation formula is:

[0099]

[0100] In addition, since the regression loss of angle θ cannot distinguish the front and rear directions of the vehicle, a cross entropy classification loss L is added dir To determine the direction of angle θ, the network can learn the front and rear directions of the vehicle.

[0101] The angle classification loss function classifies the orientation according to the orientation angle of the prediction box output by the regression head to obtain the front and back orientation of the detected target. It is calculated based on the front and back orientation of the detected target. In the specific implementation, the angle classification loss function uses the cross entropy classification loss.

[0102] The neural network structure is supervised by the total loss function to obtain a trained neural network structure; the regression residual of the trained neural network structure is the regression residual output by the regression head, which is uniquely determined; and the classification map of the current scene to be tested is the classification map output by the classification head, which changes with the input three-dimensional point cloud and the corresponding two-dimensional ring view.

[0103] The preset residual of the 3D anchor box is calculated by the 7 parameters of the real box and the 7 parameters corresponding to the 3D anchor box. That is, the preset residual of the 3D anchor box includes the residuals of the 7 parameters. The specific formula is as follows:

[0104]

[0105]

[0106] Δθ=sin(θ gt -θ a )

[0107] Among them, Δx is the preset horizontal coordinate residual, x gt and x a are the horizontal coordinates of the real box and the 3D anchor box, Δy is the preset vertical coordinate residual, y gt and a are the ordinates of the real frame and the 3D anchor frame respectively, Δz is the preset ordinate residual, z gt and z a are the vertical coordinates of the real box and the 3D anchor box, Δw is the preset box width residual, and w gt and w a are the frame widths of the real frame and the 3D anchor frame, Δl is the preset frame length residual, l gt and l a are the frame lengths of the real frame and the 3D anchor frame, Δh is the preset frame height residual, and h gt and h a are the heights of the real box and the 3D anchor box, Δθ is the preset angle residual, and θ gt and θ a are the orientation angles of the real box and the 3D anchor box respectively, and d represents the diagonal length of the 3D anchor box, satisfying

[0108] The specific training process is as follows: two NVIDIA RTX2080Ti GPUs are used for training, using the ADAM optimizer, with a momentum between 0.85 and 0.95 and a batch size of 12. The initial learning rate is set to 0.001, and the change pattern conforms to the Cyclic strategy. It gradually rises to 0.01 at the beginning of training, and then gradually decreases to 0.001×10 -4 After 80 rounds of training, the entire network has converged.

[0109] 3) Input the 3D point cloud and the corresponding 2D ring view in the verification set into the trained neural network structure, output the regression residual of the neural network structure and the classification map of the current scene to be tested, calculate the predicted 3D box based on the classification map and regression residual of the current scene to be tested, and obtain the final predicted 3D box.

[0110] In step 3), all grids whose target confidence in the classification map of the current scene to be tested is greater than the target discrimination threshold are extracted and used as the location of the detected target. The parameters of the predicted three-dimensional box of the location of the detected target are calculated according to the regression residual of the neural network structure. The calculation formula is as follows:

[0111] x=x a +da Δx′, y=y a +d a Δy′, z=z a +d a Δz′

[0112] w=w a e Δw′ , l=l a e Δl′ , h=h a e Δh′

[0113] θ=arcsin(θ a +Δθ′)

[0114] Among them, Δx′, Δy′, Δz′, Δw′, Δl′, Δh′, and Δθ′ are respectively the regression horizontal coordinate residual, regression vertical coordinate residual, regression vertical coordinate residual, regression frame width residual, regression frame length residual, regression frame height residual, and regression angle residual; x represents the horizontal coordinate of the predicted three-dimensional frame, and x represents the horizontal coordinate of the predicted three-dimensional frame. a represents the horizontal coordinate of the 3D anchor box, y represents the vertical coordinate of the predicted 3D box, a represents the ordinate of the 3D anchor box, z represents the vertical coordinate of the predicted 3D box, a represents the vertical coordinate of the 3D anchor box, w represents the box width of the predicted 3D box, and w a represents the width of the 3D anchor box, l represents the length of the predicted 3D box, l a represents the length of the 3D anchor box, h represents the height of the predicted 3D box, and h a represents the height of the 3D anchor box, θ represents the orientation angle of the predicted 3D box, and θ a represents the orientation angle of the 3D anchor box, d represents the diagonal length of the 3D anchor box, satisfying

[0115] In the validation set, the accuracy gain brought by different modules of the test network is evaluated by the average precision AP of Car detection, and the evaluation basis is the size of the 3D box intersection over union (3D IoU). If the 3D IoU between the predicted box and the true value box is greater than 0.7, the detection is considered correct. The KITTI 3D object detection dataset is divided into three levels according to the detection difficulty of the true value box samples: easy, moderate, and hard, and has corresponding annotations. The detection results of some scenes are shown in Figure 5, the 3D box detection results of the LiDAR point cloud of the scene can be displayed. The average accuracy of 3D detection of 3769 LiDAR point clouds in the statistical validation set is shown in Table 2. Compared with the simple network structure baseline, the performance of the present invention is improved by about 1%-3% after each module is added. Finally, based on the benchmark algorithm, the detection accuracy of the 3D AP index can be improved by 2.36% / 3.81% / 3.46% in Easy / Moderate / Hard difficulty respectively.

[0116] Table 2 Statistical experiments on the detection accuracy of different modules for the network 3D AP and BEV AP

[0117]

[0118] In addition, in order to explore the feature contributions of the surround view branch and the point branch in the point-surround view fusion module, a comparative experiment was conducted on the two branches and their features, and the results are shown in Table 3. It can be found that using only the surround view branch features will have a better effect than using only the point branch features. The main reason is that the surround view branch features include not only the 3D position information of the point cloud, but also the dense 2D texture information. When the features of the two branches are added to the network at the same time, it can be found that the performance is further improved, which also proves the effectiveness of the point-surround view feature fusion block proposed in the present invention.

[0119] Table 3 Comparative experiments of point-surround feature extraction module

[0120]

[0121] In order to conduct a comparative experiment on the effectiveness of the hierarchical voxel column module, it was compared with the traditional voxel column module, and the experimental results are shown in Table 4. It can be found that when the look-around branch skeleton network in the point 0 look-around feature extraction module is composed of deep residual blocks, the use of hierarchical voxel columns can improve the Moderate difficulty by 1.30% on the basis of the traditional voxel column. When the residual network block is replaced with a more powerful dense residual block, the original voxel column method cannot achieve the corresponding accuracy improvement. This shows that although more significant features are obtained by enhancing the skeleton network, it is impossible to complete sufficient feature transfer through the voxel column. When the hierarchical voxel column module proposed in the present invention is used, the accuracy is significantly improved by 3.41% compared with the original benchmark. This shows that the hierarchical voxel column module proposed in the present invention is more suitable as an intermediate module for feature transfer and compression, which can better retain 3D features and reduce the accuracy loss caused by dimensional compression.

[0122] Table 4 Comparative experiments of layered voxel column modules

[0123]

[0124] In order to conduct a comparative experiment on the effectiveness of the sparse detection head, the relevant situations of the sparse detection head are analyzed and experimented. Take the bird's-eye view of the lidar point cloud with ID '000798' in the KITTI 3D Object Detection dataset as an example, as shown in the schematic diagram Figure 6 . In the detection head with densely laid original anchor frames, the number of positive and negative samples is 155 and 321408, with a huge ratio of 1:2074. After the sparse detection head proposed in the present invention removes the pointless invalid grids, the number of positive and negative samples becomes 143 and 53403, with a ratio of 1:374. It greatly alleviates the imbalance of positive and negative samples. The specific experimental results are shown in Table 4. As shown in Table 4, the sparse detection head has an improvement of 1.70% in the average accuracy of 3D AP at the Moderate difficulty level, which fully demonstrates the superiority of the sparse detection head module.

[0125] The method of the present invention uses two NVIDIA RTX2080Ti GPUs for training and testing, and uses a neural network structure to detect the three-dimensional frame of the input laser radar point cloud. Each detection takes only 28.1ms and the frame rate reaches 36.5Hz. Therefore, the computational cost of the present invention is very small and the real-time performance is high.

[0126] It can be seen that the method of the present invention can effectively utilize the point-surround view feature extraction module to extract and fuse the features of the three-dimensional point cloud and the two-dimensional surround view; utilize the layered voxel column module to compress the 3D feature point cloud to the bird's-eye view feature; and utilize the sparse detection head to classify and regress the effective grid. The overall network structure is an end-to-end structure, which utilizes the conversion of the point cloud view to drive the conversion of features, fully exploits the advantages of different views of the point cloud, and completes the task of high-precision three-dimensional target detection. The present invention has the characteristics of high precision, low computational cost, and strong real-time performance. Compared with ordinary network structure frameworks, it has reached a leading level in both precision and real-time performance, and has significantly improved the average detection of three-dimensional target detection, while maintaining a fast frame rate of 36.5Hz, and can be efficiently applied to real-time autonomous driving perception systems.

Claims

1. A real-time 3D object detection method based on point cloud cross-view feature conversion, characterized in that: The following steps are involved: 1) Collect the 3D point cloud of the scene through the laser radar, and perform surround projection on the 3D point cloud to obtain the corresponding 2D surround view; 2) Establish a neural network structure, use the known data set and the corresponding two-dimensional ring view to train the neural network structure, set the total loss function to supervise the neural network structure, and obtain a trained neural network structure; 3) Input the 3D point cloud and the corresponding 2D ring view of the scene to be tested into the trained neural network structure, output the regression residual of the neural network structure and the classification map of the current scene to be tested, calculate the predicted 3D box based on the classification map and regression residual of the current scene to be tested, and obtain the final predicted 3D box; In the step 2), the neural network structure includes a point-surround view feature extraction module, a hierarchical voxel column module and a sparse detection head module; the input of the neural network structure is input into the point-surround view feature extraction module, the point-surround view feature extraction module is connected to the sparse detection head module after passing through the hierarchical voxel column module, and the output of the sparse detection head module is used as the output of the neural network structure; The point-surround view feature extraction module includes a surround view branch, a point branch and three point-surround view feature fusion blocks, and the surround view branch is connected to the point branch through the three point-surround view feature fusion blocks; the surround view branch includes six encoding modules and four decoding modules, and the point branch includes three 1D convolutional layers; The three-dimensional point cloud is input to the first 1D convolutional layer, and the two-dimensional surround view is input to the first encoding module. The first point cloud feature P1 output by the first 1D convolutional layer and the first surround view feature E1 output by the first encoding module are both input to the first point-surround view feature fusion block. The first point-surround view feature fusion block outputs the updated first point cloud feature and the first surround view feature respectively. The updated first point cloud feature is input to the second 1D convolutional layer. The updated first surround view feature is input to the second encoding module. The second encoding module is connected to the sixth encoding module after passing through the third encoding module, the fourth encoding module and the fifth encoding module in sequence. The second point cloud feature P2 output by the second 1D convolutional layer and the sixth surround view feature E6 output by the sixth encoding module are both input to the second In the point-surround view feature fusion block, the second point-surround view feature fusion block outputs the updated second point cloud feature and the sixth surround view feature respectively, the updated second point cloud feature is input into the third 1D convolution layer, the updated sixth surround view feature is input into the first decoding module, the first decoding module is connected to the fourth decoding module after passing through the second decoding module and the third decoding module in sequence, the third point cloud feature P3 output by the third 1D convolution layer and the tenth surround view feature D4 output by the fourth decoding module are both input into the third point-surround view feature fusion block, the third point-surround view feature fusion block outputs the updated third point cloud feature and the tenth surround view feature respectively, the updated third point cloud feature is recorded as a 3D feature point cloud and used as the output of the point-surround view feature extraction module; In the point-surround view feature fusion block, first, according to the two-dimensional surround view corresponding to the point cloud feature, each point of the point cloud feature is sampled by a bilinear interpolation method to obtain the 2D feature of the point, and the input surround view feature is used as the 3D feature of the point. The 2D feature of the point and the 3D feature of the point are cascaded and input into three 1D convolutional layers connected in sequence. The output of the last 1D convolutional layer is used as the updated point cloud feature, and the output of the last 1D convolutional layer is annularly back-projected to obtain the updated surround view feature.

2. The real-time 3D object detection method based on point cloud cross-view feature conversion according to claim 1 is characterized in that: In the step 1), a two-dimensional annular view with the pitch angle and the heading angle as the coordinate axes is established, and the pitch angle and the heading angle of each point in the three-dimensional point cloud and the origin of the laser radar coordinate system are calculated and projected into the two-dimensional annular view with the pitch angle and the heading angle as the coordinate axes to obtain the corresponding two-dimensional annular view.

3. The real-time 3D object detection method based on point cloud cross-view feature conversion according to claim 1 is characterized in that: Each encoding module is composed of a dense residual block and several convolutional layers connected in sequence. The input of each encoding module is input into the current dense residual block, and the output of the last convolutional layer is used as the output of the current encoding module. The decoding module is composed of a dense residual block and several deconvolutional layers connected in sequence. The input of each decoding module is input into the current dense residual block, and the output of the last deconvolutional layer is used as the output of the current decoding module.

4. The real-time 3D object detection method based on point cloud cross-view feature conversion according to claim 1 is characterized in that: In the hierarchical voxel column module, the 3D feature point cloud input by the point-surround view feature extraction module is first voxelized to obtain a 3D point cloud mesh feature; Then, the 3D point cloud mesh features are input into the 1D and 2D convolutional layers in turn to extract features in the height dimension and length and width dimensions, and after channel reorganization, the bird's-eye view features are obtained and output.

5. The real-time 3D object detection method based on point cloud cross-view feature conversion according to claim 1, characterized in that: The sparse detection head module includes a classification head and a regression head based on anchor frames; In the sparse detection head module, a 3D anchor frame is first laid out for each point grid in the bird's-eye view feature output by the hierarchical voxel column module. The parameters of the 3D anchor frame are determined by the position of the 3D anchor frame (x, y, z), the average size of the detection target (w mean ,l mean ,h mean ) and orientation angles, each point grid has two 3D anchor frames with different orientation angles; then the 3D anchor frames are divided into positive and negative samples according to the intersection-over-union ratio between the 3D anchor frames and the true frames; Then, based on the classification head of the anchor frame, the possibility of each point grid divided into positive and negative samples in the bird's-eye view feature being the foreground is predicted, and the predicted classification probability of each point grid is obtained. Based on the predicted classification probability of each point grid, a predicted classification probability set is obtained, and based on the predicted classification probability set, a classification map of the current scene is obtained and output; Finally, in the anchor-based regression head, the preset residual of the three-dimensional anchor frame is first set, and the anchor frame parameters are regressed for the three-dimensional anchor frames divided into positive and negative samples in the foreground grid of the current bird's-eye view feature to obtain the predicted frame, and the regression residual between the predicted frame and the true frame is calculated and output.

6. The real-time 3D object detection method based on point cloud cross-view feature conversion according to claim 5, characterized in that: The total loss function includes the 3D box classification loss function, the 3D box regression loss function and the angle classification loss function, and the calculation formula is as follows: Among them, L represents the total loss function value, L cls represents the 3D box classification loss function, L loc Represents the 3D box regression loss function value, L dir Represents the angle classification loss function value, n pos is the number of positive samples in all 3D anchor boxes, λ loc , cls , dir are the first, second, and third loss weights respectively; The three-dimensional box classification loss function is calculated based on the predicted classification probability set, the three-dimensional box regression loss function is calculated based on the preset residual and the regression residual, and the angle classification loss function is obtained by classifying the orientation according to the orientation angle of the predicted box output by the regression head to obtain the front and back orientation of the detected target, which is calculated based on the front and back orientation of the detected target.

7. The real-time 3D object detection method based on point cloud cross-view feature conversion according to claim 5, characterized in that: The preset residual of the 3D anchor frame is calculated by 7 parameters of the real frame and 7 parameters corresponding to the 3D anchor frame. The specific formula is as follows: Δθ=sin(θ gt -θ a ) Among them, Δx is the preset horizontal coordinate residual, x gt and x a are the horizontal coordinates of the real box and the 3D anchor box, Δy is the preset vertical coordinate residual, y gt and a are the ordinates of the real frame and the 3D anchor frame respectively, Δz is the preset ordinate residual, z gt and z a are the vertical coordinates of the real box and the 3D anchor box, Δw is the preset box width residual, and w gt and w a are the frame widths of the real frame and the 3D anchor frame, Δl is the preset frame length residual, l gt and l a are the frame lengths of the real frame and the 3D anchor frame, Δh is the preset frame height residual, and h gt and h a are the heights of the real box and the 3D anchor box, Δθ is the preset angle residual, and θ gt and θ a are the orientation angles of the real box and the 3D anchor box respectively, and d represents the diagonal length of the 3D anchor box, satisfying 8. The real-time 3D object detection method based on point cloud cross-view feature conversion according to claim 1 is characterized in that: In the step 3), all grids whose target confidence is greater than the target discrimination threshold in the classification map of the current scene to be tested are extracted and used as the location of the detected target, and the parameters of the predicted three-dimensional box of the location of the detected target are calculated according to the regression residual of the neural network structure. The calculation formula is as follows: x=x a +d a Δx′,y=y a +d a Δy′,z=z a +d a Δz′ w=w a have been Δw′ ,l=l a have been Δl′ ,h=h a have been Δh′ θ=arcsin(θ a +Δθ′) Among them, Δx′, Δy′, Δz′, Δw′, Δl′, Δh′, Δθ′ are respectively the regression horizontal coordinate residual, regression vertical coordinate residual, regression frame width residual, regression frame length residual, regression frame height residual, and regression angle residual; x represents the horizontal coordinate of the predicted 3D frame, x represents the horizontal coordinate of the predicted 3D frame, and x represents the vertical coordinate of the predicted 3D frame. a represents the horizontal coordinate of the 3D anchor box, y represents the vertical coordinate of the predicted 3D box, a represents the ordinate of the 3D anchor box, z represents the vertical coordinate of the predicted 3D box, a represents the vertical coordinate of the 3D anchor box, w represents the box width of the predicted 3D box, and w a represents the width of the 3D anchor box, l represents the length of the predicted 3D box, l a represents the length of the 3D anchor box, h represents the height of the predicted 3D box, and h a represents the height of the 3D anchor box, θ represents the orientation angle of the predicted 3D box, and θ a represents the orientation angle of the 3D anchor box, d represents the diagonal length of the 3D anchor box, satisfying

Images