An efficient 3D target detection method, device, terminal and storage medium

Abstract

The application discloses a kind of high-efficiency 3D target detection method, device, terminal and storage medium, the method includes: obtaining the point cloud data training set of target to be detected;Based on the point cloud data training set of the target to be detected, utilize HRNet network structure, using hierarchical refinement and dynamic sample allocation, the point cloud data training set of the target to be detected is trained, and the 3D target detection model of the target to be detected is obtained;Obtain the point cloud data detection set of the target to be detected;Using the 3D target detection model of the target to be detected, the point cloud data detection set of the target to be detected is detected, and the 3D target detection of the target to be detected is realized.The scheme, by using hierarchical refinement and dynamic sample allocation scheme, improve the accuracy of 3D target detection.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computers, and particularly relates to an efficient 3D target detection method, device, terminal and storage medium, and especially relates to an efficient multi-stage 3D target detection algorithm based on hierarchical refinement, device, terminal and storage medium. BACKGROUND

[0002] The 3D target detection technology based on point clouds is developing rapidly, and in related schemes, the two-stage detection architecture is widely studied due to its high accuracy. In the two-stage detection architecture, the second stage improves the extraction of point cloud features to improve the detection performance of the 3D target detection algorithm based on point clouds. However, the prediction of target categories and positions of the 3D target detection algorithm based on point clouds is realized by two independent branches, which makes it difficult for the 3D target detection model based on point clouds to simultaneously optimize the prediction of target categories and positions, affecting the accuracy of 3D target detection.

[0003] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0004] The present application aims to provide an efficient 3D target detection method, device, terminal and storage medium to solve the problem that the prediction of target categories and positions of the 3D target detection algorithm based on point clouds is realized by two independent branches, which makes it difficult for the 3D target detection model based on point clouds to simultaneously optimize the prediction of target categories and positions, affecting the accuracy of 3D target detection, and to achieve the effect of improving the accuracy of 3D target detection by using the scheme of hierarchical refinement and dynamic sample allocation.

[0005] The present application provides an efficient 3D target detection method, comprising: obtaining a point cloud data training set of a target to be detected; based on the point cloud data training set of the target to be detected, using an HRNet network structure, adopting hierarchical refinement and dynamic sample allocation, training the point cloud data training set of the target to be detected to obtain a 3D target detection model of the target to be detected; obtaining a point cloud data detection set of the target to be detected; using the 3D target detection model of the target to be detected to detect the point cloud data detection set of the target to be detected to realize 3D target detection of the target to be detected.

[0006] In some embodiments, based on the point cloud data training set of the target to be detected, a 3D target detection model of the target to be detected is obtained by training the point cloud data training set of the target to be detected using an HRNet network structure, adopting hierarchical refinement and dynamic sample allocation, including: based on the point cloud data training set of the target to be detected, performing one or more of point cloud preprocessing, data enhancement, object definition, point cloud voxelization, voxel encoding, and 3D feature extraction and 2D feature extraction to obtain region proposals in each of two or more stages; for the region proposals in each of the two or more stages, selecting samples in each stage, and performing hierarchical refinement network processing on the region proposals on multi-scale voxel features to obtain hierarchical refinement results; wherein in the sample selection process in each stage, the true positive samples of the corresponding stage are dynamically selected for regression to obtain selected samples in each stage; based on the hierarchical refinement results, the training is continued, and the training results obtained by each layer are further refined using features of multiple scales to finally obtain the 3D target detection model of the target to be detected.

[0007] In some embodiments, in the process of selecting samples in each of the two or more stages for the region proposals in each of the two or more stages, performing hierarchical refinement network processing on the region proposals on multi-scale voxel features to obtain hierarchical refinement results, includes: for the region proposals in each of the two or more stages, the following formula is used to perform hierarchical refinement network processing to obtain hierarchical refinement results:

[0008] Cls k ,Reg k =f k (R k-1 (Cls k-1 ,Reg k-1 )),

[0009] wherein k represents the k-th layer voxel feature, R represents a candidate box encoding function for converting a regression vector into (x, y, z, l, w, h, θ) format from the region proposal, Cls represents a classification prediction result, Reg represents a position prediction result, R represents a candidate box encoding function, f k represents a feature extraction method, i.e., a feature propagation function based on channel attention.

[0010] In some embodiments, in the sample selection process in each stage, the true positive samples of the corresponding stage are dynamically selected for regression to obtain the selected samples of each stage, including: for each candidate box in the region proposal of each stage in two or more stages, finding the closest k region proposals to each real box, where k is a hyperparameter; calculating the 3D IoU between the k candidate boxes in the closest k region proposals and the real box to obtain the 3D IoU statistical features of each real box; based on the 3D IoU statistical features of each real box, calculating the average and standard deviation of the 3D IoU of each real box and the candidate box in each frame of point cloud as the final IoU threshold to divide the positive and negative samples to obtain the selected samples of each stage; wherein in the final IoU threshold to divide the positive and negative samples, when the candidate box is closer to the real box, the threshold is dynamically changed according to the distribution of IoU, and the candidate box with IoU greater than the set threshold is selected as the positive sample for regression.

[0011] To match the above method, another aspect of the present application provides an efficient 3D target detection device, comprising: an acquisition unit configured to acquire a point cloud data training set of a target to be detected; a control unit configured to train the point cloud data training set of the target to be detected based on the point cloud data training set of the target to be detected, using an HRNet network structure, adopting a hierarchical refinement and dynamic sample allocation manner, to obtain a 3D target detection model of the target to be detected; the acquisition unit is also configured to acquire a point cloud data detection set of the target to be detected; the control unit is also configured to detect the point cloud data detection set of the target to be detected using the 3D target detection model of the target to be detected, to realize 3D target detection of the target to be detected.

[0012] In some embodiments, the control unit trains the point cloud data training set of the target to be detected based on the point cloud data training set of the target to be detected, using an HRNet network structure, adopting a hierarchical refinement and dynamic sample allocation manner, to obtain a 3D target detection model of the target to be detected, including: based on the point cloud data training set of the target to be detected, performing one or more of point cloud preprocessing, data enhancement, object definition, point cloud voxelization, voxel encoding, and 3D feature extraction and 2D feature extraction to obtain region proposals of each stage in two or more stages; for the region proposals of each stage in two or more stages, selecting samples of each stage, and performing hierarchical refinement network processing on the region proposals on multi-scale voxel features to obtain hierarchical refinement results; wherein in the sample selection process in each stage, the true positive samples of the corresponding stage are dynamically selected for regression to obtain the selected samples of each stage; based on the hierarchical refinement results, the training results of each layer are further refined using features of multiple scales to finally obtain the 3D target detection model of the target to be detected.

[0013] In some embodiments, the control unit, in the region proposal of each of the two or more stages, selects samples of each stage, and processes the region proposal on the multi-scale voxel feature through a hierarchical refinement network to obtain a hierarchical refinement result, including: for the region proposal of each of the two or more stages, processing the region proposal through a hierarchical refinement network by using the following formula to obtain a hierarchical refinement result:

[0014] Cls k ,Reg k =f k (R k-1 (Cls k-1 ,Reg k-1 )),

[0015] Wherein, k represents the k-th layer voxel feature, R represents a candidate box encoding function for converting a regression vector into an (x, y, z, l, w, h, theta) format from a region proposal, Cls represents a classification prediction result, Reg represents a position prediction result, R represents a candidate box encoding function, f k represents a feature extraction method, i.e., a channel attention-based feature propagation function.

[0016] In some embodiments, the control unit, in the sample selection process in each stage, dynamically selects true positive samples of the corresponding stage for regression to obtain selected samples of each stage, including: for the candidate boxes in the region proposal of each of the two or more stages, finding the closest k region proposals to each ground truth box, wherein k is a hyperparameter; calculating the 3D IoU between the k candidate boxes in the closest k region proposals and the ground truth box to obtain the 3D IoU statistical features of each ground truth box; based on the 3D IoU statistical features of each ground truth box, calculating the average and standard deviation of the 3D IoU of each ground truth box and the candidate boxes in each frame of point cloud as the final IoU threshold to divide positive and negative samples to obtain selected samples of each stage; wherein, in the final IoU threshold to divide positive and negative samples, when the candidate box is closer to the ground truth box, the threshold is dynamically changed according to the distribution of IoU, and the candidate box with IoU greater than the set threshold is selected as the positive sample for regression.

[0017] To match the above device, the present application further provides a terminal, including: the above-mentioned efficient 3D target detection device.

[0018] To match the above method, the present application further provides a storage medium, including a stored program, wherein when the program runs, the device where the storage medium is located executes the above-mentioned efficient 3D target detection method.

[0019] Thus, the scheme of the present application sets a high-performance multi-stage 3D target detection model based on hierarchical refinement, which includes a hierarchical refinement module and a dynamic sample allocation module, wherein the hierarchical refinement module can use multi-scale voxel features to optimize the regression branch one by one, and finally realize higher positioning accuracy; the dynamic sample allocation module can adaptively learn the intersection over union threshold of each stage positive and negative sample division to improve the accuracy of sample division; further, based on the hierarchical refinement module and the dynamic sample allocation module, a multi-stage refinement network of a high-efficiency multi-stage 3D target detection algorithm, namely HRNet network, is set, and the multi-stage refinement network is used to realize 3D target detection; thereby, the accuracy of 3D target detection is improved by using the hierarchical refinement and dynamic sample allocation scheme.

[0020] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application.

[0021] The technical scheme of the present application will be further described in detail below with the help of the accompanying drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 Distribution diagrams of different stage IoU and confidence, wherein (a) is a distribution diagram of different stage IoU, and (b) is a distribution diagram of different stage confidence;

[0023] Figure 2 Distribution diagrams of IoU and confidence after repeating two stages multiple times, wherein (a) is a distribution diagram of IoU after repeating two stages multiple times, and (b) is a distribution diagram of confidence after repeating two stages multiple times;

[0024] Figure 3 The flowchart of an embodiment of the high-efficiency 3D target detection method of the present application is shown in Figure 3;

[0025] Figure 4 The flowchart of an embodiment of the method of the present application, which uses hierarchical refinement and dynamic sample allocation to train the point cloud data training set of the target to be detected, is shown in Figure 4;

[0026] Figure 5 The flowchart of an embodiment of the method of the present application, which dynamically selects true positive samples of the corresponding stage for regression, is shown in Figure 5;

[0027] Figure 6 The structural diagram of an embodiment of the high-efficiency 3D target detection device of the present application is shown in Figure 6;

[0028] Figure 7 The diagram of the structure of the HRNet network is shown in Figure 7;

[0029] Figure 8 Table for comparison of HRNet with state-of-the-art car detection methods;

[0030] Figure 9 Effect diagram for detection effect visualization, wherein (a) is effect Figure 1 , (b) is effect Figure 2 , (c) is effect Figure 3 ;

[0031] Figure 10 Effect diagram for detection frame optimization in each stage, wherein (a) is effect diagram for detection frame optimization in the second stage, (b) is effect diagram for detection frame optimization in the second stage, and (c) is effect diagram for detection frame optimization in the second stage.

[0032] In combination with the drawings, the following is the meaning of the reference signs in the embodiments of the present application:

[0033] 102 - acquisition unit; 104 - control unit. DETAILED DESCRIPTION

[0034] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in combination with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0035] Three-dimensional object detection is considered a key task for providing environmental perception for autonomous driving and industrial robots. Point clouds obtained using LiDAR sensors can provide accurate pose information of objects, which is crucial for accurately detecting the position of objects; wherein LiDAR is a system that integrates three technologies of laser, global positioning system (GPS) and inertial navigation system (INS). Three-dimensional object detection based on point clouds aims to detect the class, size and position of objects from point cloud scenes. Due to the inherent characteristics of LiDAR sensors, the point cloud data of objects is usually incomplete due to truncation, occlusion, sparsity or different distances from the camera, which results in the point cloud being unable to provide complete context information of the object.

[0036] Object detection can be categorized into single-stage, two-stage, and multi-stage methods. Single-stage methods directly detect objects from the original point cloud scene, extracting features only once from the entire point cloud, and then classifying and regressing the object accordingly. Due to the large number of points in the point cloud scene, single-stage methods cannot design complex feature extraction schemes to improve inference efficiency, resulting in lower accuracy compared to two-stage methods. Furthermore, the original point cloud contains too much background information, negatively impacting object feature extraction. Therefore, object detection methods in related schemes first generate region proposals from the original point cloud using efficient feature extraction methods, and then further extract features from these region proposals using more complex algorithms to balance computational efficiency and accuracy. Thus, most detection methods in related schemes focus on improving two-stage feature extraction methods to achieve better detection accuracy.

[0037] While the second stage can significantly improve detection performance, previous work has rarely focused on the essential differences between the different stages. 3D object detection is essentially a task of object classification and location regression. However, the classification and regression branches are trained independently, inevitably leading to discrepancies between them. For example, many region proposals may have high confidence scores but low Intersection over Union (IoU) ratios with the ground truth boxes. An ideal 3D object detection method should ensure that region proposals with high confidence scores also have high localization accuracy. To address this issue, most detection methods in related schemes establish a link between the classification and regression branches. For example, FV2P (From Voxel to Point) proposes an IoU-guided detection framework and uses IoU instead of confidence to filter candidate boxes. AFDetV2 uses the result of multiplying the confidence score and IoU to rank candidate boxes. Cascade R-CNN, a detection method based on 2D images, is a pioneering work in multi-stage detection. CasA applies a multi-stage structure to 3D detection of point clouds through multiple iterations of the second stage.

[0038] However, the detection performance of the above-mentioned detection methods is still limited. Figure 1 The diagrams illustrate the distribution of IoU and confidence scores at different stages, where (a) shows the distribution of IoU at different stages and (b) shows the distribution of confidence scores at different stages. Since the second stage typically involves more complex feature extraction to refine the region proposals than the first stage, the confidence distribution of the candidate boxes is significantly improved in the second stage, but the improvement in IoU distribution is limited. Figure 1 (a) and Figure 1 As shown in Figure (b), the confidence scores and IoU of all candidate boxes were calculated, and the number of candidate boxes for each interval was also calculated. Figure 1In the middle (a), the horizontal coordinate IoU is the intersection over union between the detection box and the label box, the vertical coordinate box number represents the number of detection boxes, first stage IoU distribution represents the IoU distribution of the detection boxes calculated by the model after the first stage; second stage IoU distribution represents the IoU distribution of the detection boxes calculated by the model after the first and second stages. Figure 1 In the middle (b), the horizontal coordinate confidence score is the confidence of the detection box, the vertical coordinate box number represents the number of detection boxes, first stage conf distribution represents the confidence distribution curve of the detection boxes calculated by the model after the first stage; second stage conf distribution represents the confidence distribution curve calculated by the model after the first and second stages. Experiments were carried out on the KITTI dataset, and the classic single-stage detection algorithm SECOND was used as the backbone network of the first stage, and Voxel R-CNN was used as the backbone network of the second stage. All experimental configurations are consistent with CT3D and Voxel R-CNN. Among them, the KITTI dataset is jointly established by the Karlsruhe Institute of Technology in Germany and the Toyota Technical Research Institute in the United States, and is currently the largest computer vision algorithm evaluation dataset in the international automatic driving scene. This dataset is used to evaluate the performance of computer vision technologies such as stereo, optical flow, visual odometry, 3D object detection, and 3D tracking in a vehicle environment.

[0039] The median value of the confidence distribution and the IoU distribution show inconsistent behavior, where the median value of the confidence distribution refers to half of the total number of predicted boxes. Specifically, the second stage is more effective for the confidence branch. When the distribution of the regression target is unstable due to sample selection, the confidence score may decrease. Figure 2 The distribution of IoU and confidence after repeating the second stage multiple times is shown in (a) and (b). In another experiment, if the second stage is repeatedly performed, the IoU distribution changes little, but the overall confidence score decreases significantly, as shown in Figure 2 In the middle (a) and Figure 2 (b), the experimental settings are the same as Figure 1 The reason for this difference is that the classification branch and the localization branch have different optimization paradigms. Among them, Figure 2In the middle (a), the horizontal coordinate IoU is the intersection over union between the detection box and the label box, the vertical coordinate box number represents the number of detection boxes, and the second stage IoU distribution represents the IoU distribution of the detection boxes calculated by the model after the second stage. The repeat second stage twice IoU distribution represents the IoU distribution of the detection boxes calculated by the model after repeating the second stage twice. Figure 2 In the middle (b), the horizontal coordinate confidence score is the confidence of the detection box, the vertical coordinate box number represents the number of detection boxes, and the second stage conf distribution represents the confidence distribution curve of the detection boxes calculated by the model after the second stage. The repeat second stage twice conf distribution represents the confidence distribution curve calculated by the model after repeating the second stage twice. Generally, the regression branch needs to predict the size, position and direction angle of the object, while the classification branch focuses on the prominent part of the object, rather than the integral of the whole object. In particular, when the prediction result gradually approaches the true value, the fixed IoU threshold will cause more inaccurate samples to be selected, thereby causing ambiguity in the calculation of the regression loss.

[0040] Inspired by the above observations, the applicant believes it is necessary to establish a multi-stage detector that can iteratively refine the candidate boxes, gradually optimize the classification and regression branches, and dynamically select samples for the calculation of the regression loss at each stage.

[0041] Considering that in point cloud-based 3D object detection algorithms, the prediction of object category and location is achieved through two independent branches, it is difficult for point cloud-based 3D object detection models to simultaneously optimize the prediction of object category and location, affecting the accuracy of 3D object detection. Furthermore, in these algorithms, the regression branch needs to predict more variables than the classification branch, resulting in a speed difference between the optimization of object category prediction and location prediction, which also affects the accuracy of 3D object detection. Additionally, point cloud-based 3D object detection algorithms typically use a fixed intersection-union (IU) threshold to distinguish between positive and negative samples. For multi-stage detection algorithms, since the distribution of the IU threshold changes at each stage, a fixed threshold cannot adequately distinguish the samples, further impacting the accuracy of 3D object detection. Therefore, the present invention proposes an efficient 3D target detection method, specifically a high-performance multi-stage 3D target detection algorithm based on hierarchical refinement, which improves the accuracy of 3D target detection by adopting a hierarchical refinement and dynamic sample allocation scheme.

[0042] According to embodiments of the present invention, an efficient 3D target detection method is provided, such as... Figure 3 The diagram shows a flowchart of an embodiment of the method of the present invention. This efficient 3D target detection method may include steps S110 to S140.

[0043] In step S110, a training set of point cloud data for the target to be detected is obtained. Point cloud data refers to a set of vectors in a three-dimensional coordinate system. The scanned data is recorded in the form of points, each point containing three-dimensional coordinates, and some may contain color information (RGB) or reflectance intensity information. The target to be detected can be one or more of the following: a car, a pedestrian, or a bicycle.

[0044] In step S120, based on the point cloud data training set of the target to be detected, the HRNet network structure is used to train the point cloud data training set of the target to be detected by employing hierarchical refinement and dynamic sample allocation, thereby obtaining the 3D target detection model of the target to be detected.

[0045] In step S130, a point cloud data detection set of the target to be detected is obtained. The point cloud data detection set of the target to be detected may be either on-site measurement data of the target or a collection of on-site measurement data.

[0046] At step S140, the point cloud data detection set of the to-be-detected target is detected by using the 3D target detection model of the to-be-detected target, and 3D target detection of the to-be-detected target is realized.

[0047] In the scheme of the present application, a high-performance multi-stage 3D target detection algorithm based on hierarchical refinement is proposed, which sets a high-performance multi-stage 3D target detection model based on hierarchical refinement, which includes a hierarchical refinement (Hierarchical Refinement) module and a dynamic sample allocation (Dynamic Sample Selection) module. The hierarchical refinement module can use multi-scale voxel features to optimize the regression branch one by one, and finally achieve high positioning accuracy. The dynamic sample allocation module can adaptively learn the intersection over union threshold of each stage of positive and negative sample division to improve the accuracy of sample division, thereby improving the overall performance of the 3D target detection model. Furthermore, in the high-performance multi-stage 3D target detection model based on hierarchical refinement, the hierarchical refinement module and the dynamic sample allocation module are set, and a multi-stage refinement network of the high-efficiency multi-stage 3D target detection algorithm, i.e. HRNet (Hierarchical Refinement Network) network, is proposed. The use of the multi-stage refinement network for 3D target detection can at least solve the problem that the 3D target detection model based on point cloud is difficult to simultaneously optimize the prediction of target category and position, thereby affecting the accuracy of 3D target detection, and is conducive to improving the accuracy of 3D target detection.

[0048] Specifically, in the scheme of the present application, the HRNet network structure comprises an RPN network module, an HRN network module and an SAP module, wherein the RPN network module is a region proposal network (RPN) module, the HRN network module is a hierarchical refinement network (HRN) module, and the SAP module is a scale-aware attentive propagation (SAP) module. The RPN network module is configured to generate region proposals of each of two or more stages based on the point cloud data training set of the target to be detected. The HRN network module is configured to perform hierarchical refinement network and dynamic sample allocation processing on the region proposals of each of the two or more stages to obtain hierarchical refinement results. Specifically, the HRN network module comprises a hierarchical refinement module and a dynamic sample selection module (DSS module). The hierarchical refinement module is configured to select samples of each stage for the region proposals of each of the two or more stages, and perform hierarchical refinement on the region proposals in multi-scale voxel features to obtain hierarchical refinement results. The dynamic sample selection module is configured to dynamically select true positive samples of the corresponding stage for regression in the sample selection process of each stage. The SAP module is configured to be trained based on the hierarchical refinement results, and perform feature propagation between different scale features based on the training results to achieve training of the point cloud data training set of the target to be detected in a hierarchical refinement and dynamic sample allocation manner, and obtain a 3D target detection model of the target to be detected.

[0049] In some embodiments, the specific process of training the point cloud data training set of the target to be detected in step S120 by using the HRNet network structure in a hierarchical refinement and dynamic sample allocation manner to obtain the 3D target detection model of the target to be detected is described in the following example.

[0050] The following will be described in combination with Figure 4 The following will be described in combination with

[0051] Step S210, based on the point cloud data training set of the to-be-detected target, one or more of the following processes are performed: point cloud preprocessing, data enhancement, object definition, point cloud voxelization, voxel encoding, 3D feature extraction, and 2D feature extraction, to obtain a region proposal of each of the two or more stages.

[0052] Step S220, for the region proposal of each of the two or more stages, a sample of each stage is selected, and a hierarchical refinement network is used to process the region proposal on the multi-scale voxel features to obtain a hierarchical refinement result. In the sample selection process in each stage, the true positive samples of the corresponding stage are dynamically selected for regression to obtain the selected samples of each stage.

[0053] Step S230, based on the hierarchical refinement result, the training is continued, and the training results of each layer are further refined using features of multiple scales to finally obtain a 3D target detection model of the to-be-detected target.

[0054] In the scheme of the present application, a HRNet (Hierarchical Refinement Network) network with hierarchical refinement capability is proposed to achieve better detection performance. Specifically, the scheme of the present application performs hierarchical refinement on the region proposal (i.e., region proposal) on multi-scale voxel features to gradually and effectively improve the performance of the regression branch. The scheme of the present application proposes a Scale-aware Attentive propagation (SAP) module to propagate features between different scale features to utilize more spatial information for object detection. In order to improve the sample selection of each stage, the scheme of the present application proposes a Dynamic Sample Selection (DSS) module to select more true positive samples for regression.

[0055] In some embodiments, the step S120 of selecting a sample of each stage on the region proposal of each of the two or more stages and performing hierarchical refinement network processing on the region proposal on the multi-scale voxel features to obtain a hierarchical refinement result comprises: for the region proposal of each of the two or more stages, the hierarchical refinement network processing is performed using the following formula to obtain a hierarchical refinement result:

[0056] Cls k ,Reg k =f k (R k-1 (Cls k-1 ,Reg k-1 )),

[0057] Wherein, k represents the kth layer voxel feature, R represents a candidate box encoding function, which is used to convert the regression vector into (x, y, z, l, w, h, theta) format from the region proposal, Cls represents the classification prediction result, Reg represents the position prediction result, R represents the candidate box encoding function, f k represents a feature extraction method, i.e., a channel attention based feature propagation function.

[0058] Specifically, the scheme of the present application proposes a high-performance multi-stage 3D target detection algorithm based on hierarchical refinement, which can include the following steps 1 to step 10.

[0059] Step 1, point cloud preprocessing stage.

[0060] Define the point cloud as Wherein, p i ={x, y, z, r} represents a point in the point cloud, x, y, z represents the three-dimensional coordinates of the point, r represents the reflectivity of the point, i represents the index of the point in the point cloud, n represents the number of points in the point cloud, and i, n are both positive integers. First, the point cloud is voxelized, represented as Wherein, L, W, H are the number of voxels in the x, y, z directions. In the point cloud preprocessing stage, the input point cloud is first filtered. To limit the size of the point cloud sampling space, only points within the specified range (x min ,x max ), (y min ,y max ), (z min ,z max ) along the x, y, z directions are retained, and points outside the range are discarded. Then, foreground objects with a point number less than the threshold t are filtered. Objects with too few reflection points are considered to contain insufficient information to express the spatial geometric information of the object, making it difficult to identify them, so these objects will not be marked as true values.

[0061] Step 2, data augmentation, which can include the following steps 21 to step 24.

[0062] Step 21, increase the detected target: store all scene true values in a dictionary, and randomly select true values from the dictionary to add to the current point cloud during training, which increases the number of targets per frame of point cloud and simulates objects in different environments. When adding, physical collision test is performed to remove sampling objects that collide with other objects.

[0063] Step 22, global rotation and scaling: randomly scale all points in the point cloud by a factor in the range of [0.95, 1.05], and randomly rotate the points by an angle in the range of .

[0064] Step 23, true value rotation and linear transformation: random linear transformation obeying Gaussian distribution is performed on all real boxes and points in real boxes, and random angle rotation is performed in the range of to simulate real values and steering of different sizes.

[0065] Step 24, flip: the point cloud is randomly flipped along the x axis.

[0066] Step 3, Anchor definition (i.e. object definition).

[0067] The scheme of the application uniformly trains three categories of cars, pedestrians and bicycles, wherein the anchor box size of the car is set to 3.90m, 1.60m and 1.56m in the x, y and z directions respectively, the anchor box size of the pedestrian is set to 0.80m, 0.60m and 1.73m in the x, y and z directions respectively, and the anchor box size of the bicycle is set to 1.76m, 0.60m and 1.73m in the x, y and z directions respectively. For the positive and negative sample division of the car category, if the IoU between the anchor box predicted by the model and the real value is greater than 0.60, it is divided into a positive sample, and if it is less than 0.45, it is divided into a negative sample, and the predicted anchor box between 0.45 and 0.60 is discarded. For pedestrians and bicycles, if the IoU between the anchor box predicted by the model and the real value is greater than 0.50, it is divided into a positive sample, and if it is less than 0.35, it is divided into a negative sample, and the predicted anchor box between 0.35 and 0.50 is discarded.

[0068] Step 4, point cloud voxelization stage.

[0069] The point cloud has the characteristics of disorder, irregularity and permutation invariance, and cannot be used for feature extraction by convolution operation in related schemes. The voxel method first divides the irregular point cloud space into uniformly stacked voxels, and then the model processes the point cloud with voxels as the smallest unit, and extracts features of the point cloud by 3D convolution, which has higher efficiency than directly processing points. First, the point cloud is equally spaced, and the size of each voxel in the x, y and z directions is defined as V x , V y , V z , respectively, and the number of voxel grids in the x, y and z directions is where D, W and H are the number of voxel grids in the depth, width and height directions, respectively. In the training stage, the number of voxel grids is limited to a maximum of 16,000.

[0070] Given the original point cloud p n = (x n , y n , z n , r n ), wherein xn , y n , z n are the spatial coordinates of the point, r n is the reflectivity information of the point. After filtering, the points are put into the corresponding voxel grid, then the point p n The corresponding voxel grid coordinates are

[0071] For data alignment, the point cloud has a value range of [0.0, 70.4] m in the x-axis direction, [-40.0, 40.0] m in the y-axis direction, and [-3.0, 1.0] m in the z-axis direction. The voxel block size in the x, y, and z directions is 0.05 m, 0.05 m, and 0.10 m, respectively, and the number of voxel blocks in the three directions is 1 408, 1 600, and 40, respectively. The number of points in each voxel is set to not more than 5.

[0072] Step 5, voxel encoding stage.

[0073] First, define the non-empty voxel where i = 1,..., n. For each voxel V, calculate the average coordinate value of the points it contains as the centroid (v x , v y , v z ), use the distance from the point to the centroid to enhance the features of each point, and add the reflectivity r i as supplementary information to obtain where V in is the voxel feature, i = 1,..., n. Then, V in is converted to an m-dimensional feature space by PointNet, denoted as Then, the maximum pooling function is used to aggregate f i to obtain the feature of the voxel V

[0074] Step 6, 3D feature extraction network.

[0075] 3D sparse convolution and sub-manifold convolution are used for multi-scale feature extraction of voxels, and convolution operation is only performed on non-empty voxels. Down-sampling by 1x, 2x, 4x, and 8x is performed respectively, and the last layer of features is compressed on a two-dimensional plane to generate region proposals.

[0076] Step 7, 2D feature extraction network.

[0077] The 2D backbone network uses a convolutional neural network with a feature pyramid structure to extract multi-scale features, including two channels: the first channel keeps the feature map resolution size unchanged and performs channel transformation in the feature dimension. The second channel keeps the feature dimension unchanged and reduces the feature map resolution size by half for feature extraction. Finally, the features of the two channels are spliced and input into the detection head to generate region proposals.

[0078] Step 8, hierarchical refinement network.

[0079] Most two-stage detectors in related schemes refine the candidate box through fine-grained features. The candidate box is first sorted by classification score, and then non-maximum suppression is performed through the IoU threshold. Compared with the classification branch, the regression branch needs to predict more variables, and the classification branch can usually approach the target true label faster. Inspired by Cascaded RCNN, in two-dimensional detection, the applicant found that performing multi-stage refinement can improve performance. However, the computational cost is difficult to bear, which makes it difficult for the multi-stage method to be further applied in practice. Therefore, the scheme of the present application introduces a hierarchical refinement module to perform multi-stage refinement of the candidate box with higher efficiency.

[0080] Among them, hierarchical refinement is to optimize the position of the candidate in a top-down manner on multi-scale features, thereby continuously improving the accuracy of the candidate box. And through the attention mechanism between different scale features, the confidence of the candidate box is improved. In this way, the combination of optimizing the position of the candidate and propagating features between different scale features through the attention mechanism can improve the positioning accuracy and confidence at the same time, thereby significantly improving the detection accuracy. In each stage of target detection, the single-stage speed is the fastest, but the accuracy is the lowest, the double-stage speed is slightly slower than the single-stage, but the accuracy is high, and the multi-stage accuracy is the highest, but the speed is the slowest. The stage of target detection involved in the scheme of the present application belongs to the multi-stage category, and the hierarchical refinement scheme is used to achieve the same speed as the double-stage, and even faster, while having very high accuracy.

[0081] The scheme of the present application proposes a hierarchical refinement network to gradually refine the proposals generated by SECOND. Specifically, region proposals are generated at each scale of the voxel layer, and new region proposals are further refined by higher resolution voxel features. Then, the IoU threshold is dynamically recalculated at each stage to obtain more true positive samples. It is worth mentioning that the hierarchical refinement network is more efficient without increasing too much cost. It essentially uses higher scale object position information and more rich semantic and background information to guide the positioning of the candidate box, thereby propagating the object position information from lower resolution to higher resolution to obtain higher precision.

[0082] Layered refinement is based on the multi-scale voxel features of the three-dimensional backbone. Figure 7 This is a schematic diagram of the HRNet network structure. Figure 7 In this diagram, Block_1 to Block_4 represent voxel features at different scales, BEV represents features from the BEV perspective, RegionProposal Network is the region proposal network, Scale-aware Attentive Propagation (SAP) is the scale-based feature attention propagation module, Hierarchical Refinement Network is the hierarchical refinement network, FirstStage to Forth Stage are the four candidate box refinement stages, DSS is the dynamic sample selection module, used to dynamically select positive samples from each stage for calculating regression loss, Proposal represents the candidate boxes generated at each stage, Reg and Cls are the position regression vector and confidence score, respectively, Pool Feature is the feature pooling module, Position Embedding is the point cloud position encoding, Pre-channel is the feature output from the previous stage, Add represents the vector addition operation, and ReLU is the activation function.

[0083] like Figure 7 As shown, each scale voxel feature is used to refine the region proposal. The high-performance multi-stage 3D target detection model based on hierarchical refinement proposed in this invention has a total of 4 region proposal modules.

[0084] exist Figure 7 In the example shown, the Region Proposal Network (RPN) is used to generate candidate regions, and the hierarchical refinement module, or Hierarchical Refinement Network, is used to implement hierarchical refinement. The hierarchical refinement module can be represented as:

[0085] Cls k ,Reg k =f k (R k-1 (Cls k-1 ,Reg k-1 )),

[0086] Where k represents the voxel feature of the k-th layer, R represents the candidate box encoding function, used to convert the regression vector from the region proposal into the (x,y,z,l,w,h,θ) format, and f n This refers to a feature extraction method. In the scheme of this invention, f nScale-aware Attentive Propagation (SAP) module. As we mentioned before, as Figure 2 As shown in (b), iterative refinement can gradually reduce the classification score. This is caused by the loss of context information at different stages. Therefore, the scheme of the present application proposes a SAP module to gradually aggregate context information from top to bottom in order to make full use of the features at each scale. Each regressor receives features and region proposals from the previous layer as input for further refinement.

[0087] Inspired by CT3D (i.e., a two-stage 3D object detection framework), the applicant observed that channel-wise Transformer (i.e., attention-based Transformer) can calculate attention between different channels, which can avoid aggregating similar or redundant information and integrate context information. Therefore, in the scheme of the present application, a channel-wise Transformer is used at each stage to obtain fine-grained feature representation for feature propagation. Specifically, in the scheme of the present application, each scale feature is encoded, and then a parameter-free skip connection is established between layers to enrich the features. New region proposals are propagated through the SAP module, as shown in Figure 7 The formula of the SAP module is as follows:

[0088] X m =A(T m (X m-1 ,V m )+X m-1 ),

[0089] where X m represents the output of the mth SAP module, A represents an activation function, T m represents a channel-wise Transformer for fusing context information between different channels, and V m represents the voxel features at the mth layer. In the scheme of the present application, there are no shared parameters in the refinement process at each stage, which makes the learning of different scale features more flexible.

[0090] Unlike image-based object detection, most of the spatial voxels of point clouds are sparse and do not contain any points. Low-scale high-resolution voxels contain more empty voxels than high-scale low-resolution voxels. In order to gradually optimize positioning, the refinement process starts from high-scale features, and at each refinement stage we expand each region proposal to expand the receptive field to capture more semantic and background information.

[0091] In some embodiments, in the sample selection process in each stage in step S120, the true positive samples of the corresponding stage are dynamically selected for regression, and the specific process of selecting samples in each stage is obtained, see the following exemplary description.

[0092] The following will be described in conjunction with Figure 5 An embodiment flowchart of the method of the present application is shown in the figure, which further illustrates the specific process of dynamically selecting true positive samples of the corresponding stage for regression in step S120, including steps S310 to S330.

[0093] In step S310, for each stage in the two or more stages, find the closest k region proposals to each ground truth box in the candidate boxes of the region proposal, where k is a hyperparameter.

[0094] In step S320, calculate the 3D IoU between the k candidate boxes in the closest k region proposals and the ground truth box, to obtain the 3D IoU statistical features of each ground truth box.

[0095] In step S330, based on the 3D IoU statistical features of each ground truth box, calculate the average and standard deviation of the 3D IoU between each ground truth box and the candidate box in each frame of point cloud as the final IoU threshold to divide the positive and negative samples, to obtain the selected samples in each stage. Wherein, in the final IoU threshold to divide the positive and negative samples, when the candidate box is closer to the ground truth box, the threshold is dynamically changed according to the distribution of IoU, and the candidate box with IoU greater than the set threshold is selected as the positive sample for regression.

[0096] Further, the high-performance multi-stage 3D target detection algorithm based on hierarchical refinement proposed by the scheme of the present application can further include the following steps 9 to 10.

[0097] Step 9, dynamic sample allocation.

[0098] During training, only positive samples (i.e. positive samples) are used for the calculation of the regression loss, and a fixed IoU threshold is used in the related scheme to divide positive and negative samples. When performing hierarchical refinement, the candidate frame is closer to the ground truth at each stage, which causes the distribution of IoU to change, and a fixed IoU threshold cannot effectively select samples. In order to adapt to the change of IoU distribution caused by inaccurate network parameters, different densities of voxel centers, and different distances between objects and sensors, the scheme of the present application proposes a dynamic sample selection (DSS) module, which can use the statistical characteristics of IoU to adaptively select 3D samples, so that the high-performance multi-stage 3D target detection model based on hierarchical refinement can recalculate the IoU threshold at each refinement stage to select different positive and negative samples, making the selection of positive and negative samples simple and effective without additional calculation.

[0099] Dynamic sample allocation is because when training the model, the target position needs to be detected, some frames (i.e. candidate frames) are set in advance, which need to be divided into positive samples (i.e. samples with high coincidence degree with the target) and negative samples (i.e. samples with low coincidence degree with the target), and then the positive samples are used to calculate the regression loss of the model (to solve the positioning problem of the frame). The method of dividing positive and negative samples in the related scheme is to divide the positive and negative samples based on a fixed threshold (for example, 0.5), but for a multi-stage target detection method, each stage will generate more accurate frames (i.e. candidate frames), and in theory this threshold should be increased as the frame position becomes more accurate, so if a fixed threshold is used, the quality of the positive samples will be reduced. In order to obtain higher quality frames, the scheme of the present application uses the calculation method of the average value and the variance to determine the threshold for sample division. Specifically, the algorithm generates multiple candidate frames on each target, then calculates the coincidence degree (i.e. intersection over union) of these candidate frames and the target, calculates the average value and variance of the coincidence degree of all candidate frames, and uses the average value and variance as the threshold for dividing positive and negative samples. In this way, positive and negative samples can be allocated according to different thresholds at each stage, thereby significantly improving the detection accuracy.

[0100] In general, the number of proposals (i.e. proposals) is usually much larger than the ground truth, where proposals represent candidate frames generated by the network. Therefore, the scheme of the present application finds the k regions closest to the real frame G in distance d k , where k is a hyperparameter. The distance d is calculated by the center points of the region proposal and the real frame, and the calculation formula is:

[0101]

[0102] Where x, y, z represent the center of the box, subscript p represents the region proposal, and g represents the real box.

[0103] Before performing non-maximum suppression, each real box corresponds to multiple candidate boxes. In the scheme of the present application, the 3D IoU between the nearest k candidate boxes and the real box is first calculated to obtain the 3D IoU statistical feature of each real box. Then, the average and standard deviation of the 3D IoU of each real box and the candidate box in each frame of point cloud are calculated, that is, I m and I s The DSS module can learn a specific IoU threshold for each real box. Then, I = I m + I s is taken as the final IoU threshold to divide the positive and negative samples. When the candidate box is closer to the real box, the threshold can be dynamically changed according to the distribution of IoU. Finally, the candidate box with IoU greater than I is selected as the positive sample for regression. Where I m is the mean of IoU, and I s is the variance of IoU. The variance is the variance of the IoU of the current real box and the multiple candidate boxes calculated by the network. The mean is the mean of the IoU of the current real box and the multiple candidate boxes calculated by the network. I is the final sample selection threshold.

[0104] The purpose of introducing the IoU statistical feature for sample selection is that if the quality of the candidate box is very high, then the IoU of most candidate boxes is greater than the average I m . This is unreasonable for selecting candidate boxes with different quality of IoU. For those containing objects with sparse points, the quality of the candidate box is usually low, and the IoU is also uneven, and a larger IoU threshold makes it difficult for the model to pay attention to these difficult samples. Therefore, I m is used to adjust the candidate boxes with different quality of IoU. In addition, the candidate boxes of objects with sparse points often have great differences in position. I s is used to increase the threshold to improve the quality of the positive sample. This makes the sample selection for objects with different sparsity more balanced.

[0105] Step 10, training loss.

[0106] In the scheme of the present application, the loss of the HRNet network is a multi-task loss composed of the RPN loss L rpn and the hierarchical refinement loss L HRi The network is trained in an end-to-end manner without any pre-training operation. The calculation method of the total loss L total of the HRNet network is:

[0107]

[0108] Here, i represents the i-th hierarchical refinement stage. Specifically, HRNet contains 3 refinement stages.

[0109] The RPN detector head consists of two fully connected layers and contains a nonlinear activation function. The RPN loss L... rpn Includes confidence prediction loss L cls and regression loss L reg , can be represented as:

[0110]

[0111] Among them, c g and r g Indicates the classification labels and regression target. and These represent predicted values. L cls Use the FocalLoss loss function to balance the contribution of positive and negative samples to the loss:

[0112]

[0113] Here, α and γ are the hyperparameters of Focal Loss, which are set to 0.25 and 2 respectively during the training phase.

[0114] L reg Using the Smooth-L1 loss function, only the loss on positive samples is calculated:

[0115]

[0116] Where p represents the number of positive samples. The total L rpn The loss is calculated as follows:

[0117]

[0118] The calculation and L rpn Similarly, the latter is composed of classification loss. and regression loss composition. Using the cross-entropy loss function, the calculation formula is as follows:

[0119]

[0120] in, The confidence score for stage i is represented by the cross-entropy loss function used in this invention to calculate the confidence score. Then, the losses from all stages are summed to obtain the final classification loss. The regression loss at each stage is obtained by summing the regression losses at each stage, and the sum of the regression losses at each stage is L. rpn L inreg They are the same.

[0121] The effectiveness of the proposed model was validated using the KITTI dataset. The model was tested on the test set using the official KITTI evaluation metrics, and the accuracy was calculated using 40 recall locations. Figure 8 This table compares HRNet with state-of-the-art vehicle detection methods. Experimental results on the KITTI test set are shown below. Figure 8 The comparison table shown compares HRNet with state-of-the-art car detection methods, with accuracy calculated using 40 recall locations. Extensive experiments on KITTI demonstrate that the proposed high-performance multi-stage 3D object detection model based on hierarchical refinement and its corresponding algorithm achieve high detection accuracy, particularly for car detection, reaching an advanced level. KITTI is one of the most important test sets in the field of autonomous driving, primarily targeting image processing techniques for autonomous driving, mainly applied to perception and prediction. Experimental results show that the scale-aware attention propagation module (such as the hierarchical refinement module) and dynamic sample selection module proposed in this invention significantly improve detection performance. The HRNet network in this invention outperforms other advanced detectors with significantly higher performance.

[0122] Figure 9 This is a visualization of the detection effect, where (a) shows the effect. Figure 1 (b) represents the effect. Figure 2 (c) represents the effect. Figure 3 The detection visualization effect of the solution of the present invention is as follows: Figure 9 As shown. Figure 9 The three detection results images in the image represent the detection performance under three complex environments, demonstrating that the model performs well under different conditions and exhibits strong robustness. Each image includes a camera image and a point cloud image for that environment. In the point cloud image, a first-defined color box (such as a green box) represents the car detected by the model.

[0123] Figure 10 The diagrams illustrate the optimization results of the detection boxes at each stage. (a) shows the optimization result of the detection boxes in the second stage; (b) shows the optimization result of the detection boxes in the second stage; and (c) shows the optimization result of the detection boxes in the second stage. Figure 10 In the diagram, the second set color box (e.g., a blue box) represents the target's ground truth bounding box, and the first set color box (e.g., a green box) represents the predicted bounding box generated by the model. The optimization effects of each stage of the present invention on the detection performance are as follows: Figure 10 As shown.

[0124] In the scheme of the present application, the intersection over union of the candidate box and the target is specifically calculated, while the ATSS (adaptive training sample selection) based on image detection in the related scheme calculates the intersection over union of the anchor (i.e. the objects for calculating the intersection over union are different) and the target, which makes the starting points of the two different. ATSS is only used for single-stage sample selection, while the DSS proposed in the scheme of the present application is for multi-stage sample selection. In addition, the scheme of the present application uses the distance of the 3D box center point, while ATSS uses the distance of the two-dimensional coordinate, because ATSS is based on image and is two-dimensional; while the point cloud is three-dimensional, and naturally uses three-dimensional coordinates to calculate the distance. In addition, in the scheme of the present application, the point cloud is three-dimensional coordinates, which integrates dynamic sample selection into each detection stage, while ATSS only performs threshold assignment once. The target detection method in the related scheme mostly stays at the level of improving feature extraction methods, and such methods cannot obtain better detection effect. The method proposed in the scheme of the present application combines hierarchical refinement and dynamic sample assignment, which can greatly improve the detection accuracy and maintain a fast detection speed.

[0125] Some schemes propose a dynamic environment three-dimensional reconstruction method, but the application field of this method is three-dimensional reconstruction, not the three-dimensional target detection involved in the scheme of the present application. Three-dimensional target detection is used for automatic driving environment perception in road scene, and three-dimensional reconstruction is modeling of the environment, which are not comparable. In addition, this method uses camera data of RGB, and the scheme of the present application uses point cloud data of lidar, which are obviously different in data acquisition method, so that the detection result and detection speed are also different.

[0126] Some other schemes propose a point cloud efficient 3D target detection method based on key point multi-scale feature fusion, which is a two-stage voxel method three-dimensional target detection. Compared with the scheme of the present application: the scheme of the present application is a multi-stage three-dimensional target detection method, which has higher accuracy. In the positive and negative sample assignment in the training process, the scheme of the present application proposes a dynamic sample sampling method, which can select positive samples according to different sample division thresholds in different stages, while the method uses fixed threshold sampling, which limits the detection performance. The method positions the three-dimensional box through the feature key point branch and classifies the three-dimensional box through the distance key point, while the scheme of the present application uses voxel features at different scales to optimize the candidate box positioning multiple times, and obtains more accurate detection effect. In addition, in terms of advancement, the method proposed in the scheme of the present application has higher accuracy and faster detection speed. The method extracts point cloud features through multiple downsampling, and only uses the features of the last layer for detection, while the multi-stage method proposed in the scheme of the present application uses multiple layers of voxel features to perform hierarchical optimization of the candidate box.

[0127] There are also some solutions that propose a 3D target detection algorithm based on camera and laser radar data fusion, which is a method for detecting by fusing multi-modal data, and the method proposed in the solution of the application only uses point cloud data of laser radar for detection, and better detection effect is obtained. The method first detects through the image to roughly find the position of the target in the image, and then maps it to the point cloud scene for further detection, while the method proposed in the solution of the application directly detects from the point cloud, which is faster and more accurate. However, multi-modal data fusion is not good, and in actual research, it is found that a single-mode detector based on laser radar can obtain high enough detection performance, which surpasses most multi-modal data fusion detectors.

[0128] There are also some solutions that propose an efficient 3D target detection device and method based on multi-frame point cloud data, which is a method for detecting based on multi-frame point cloud data, which is relatively complex, while the method proposed in the solution of the application is based on single-frame point cloud data for detection. The method tries to align multiple frames of point cloud to increase the number of points contained in the target, thereby improving the detection accuracy, which will bring additional calculation cost of multi-frame point cloud alignment, and the solution of the application performs multi-stage detection through multi-scale feature fusion, so that the model can learn the features of the target point cloud when it is sparse, thereby improving the robustness of the detector.

[0129] There are also some solutions that propose a 3D target detection algorithm based on multiple sensors, which is a three-dimensional target detection algorithm based on multi-modal fusion, while the solution of the application only uses point cloud data of laser radar for detection.

[0130] The technical scheme of the embodiment adopts a high-performance multi-stage 3D target detection model based on hierarchical refinement, which includes a hierarchical refinement module and a dynamic sample distribution module. The hierarchical refinement module can use multi-scale voxel features to optimize the regression branch one by one, and finally achieve high positioning accuracy. The dynamic sample distribution module can adaptively learn the intersection over union threshold of each stage of positive and negative sample division to improve the accuracy of sample division. Furthermore, based on the hierarchical refinement module and the dynamic sample distribution module, a multi-stage refinement network of the high-efficiency multi-stage 3D target detection algorithm, i.e. the HRNet network, is set up, and the 3D target detection is realized by using the multi-stage refinement network. Thus, the accuracy of 3D target detection is improved by using the hierarchical refinement and dynamic sample distribution scheme.

[0131] According to the embodiment of the application, an efficient 3D target detection device corresponding to the efficient 3D target detection method is also provided. Referring to Figure 6 the structure schematic diagram of an embodiment of the device of the application. The efficient 3D target detection device can include an acquisition unit 102 and a control unit 104.

[0132] The acquisition unit 102 is configured to acquire a point cloud data training set of a to-be-detected target. The point cloud data refers to a set of vectors in a three-dimensional coordinate system. The scanning data is recorded in the form of points, and each point contains three-dimensional coordinates, and some may contain color information (RGB) or intensity information (Intensity). The to-be-detected target can be one or more of a car, a pedestrian, and a bicycle. For specific functions and processing of the acquisition unit 102, see step S110.

[0133] The control unit 104 is configured to train the point cloud data training set of the to-be-detected target by using an HRNet network structure and a hierarchical refinement and dynamic sample allocation manner based on the point cloud data training set of the to-be-detected target, to obtain a 3D target detection model of the to-be-detected target. For specific functions and processing of the control unit 104, see step S120.

[0134] The acquisition unit 102 is further configured to acquire a point cloud data detection set of the to-be-detected target. The point cloud data detection set of the to-be-detected target can be field measurement data or a set of field measurement data of the to-be-detected target. For specific functions and processing of the acquisition unit 102, see step S130.

[0135] The control unit 104 is further configured to detect the point cloud data detection set of the to-be-detected target by using the 3D target detection model of the to-be-detected target, to realize 3D target detection of the to-be-detected target. For specific functions and processing of the control unit 104, see step S140.

[0136] In the scheme of the present application, a high-performance multi-stage 3D target detection algorithm based on hierarchical refinement is proposed. A high-performance multi-stage 3D target detection model based on hierarchical refinement is set up, which includes a hierarchical refinement (Hierarchical Refinement) module and a dynamic sample allocation (Dynamic Sample Selection) module. The hierarchical refinement module can use multi-scale voxel features to optimize the regression branch one by one, and finally achieve high positioning accuracy. The dynamic sample allocation module can adaptively learn the intersection over union threshold of each stage of positive and negative sample division to improve the accuracy of sample division, thereby improving the overall performance of the 3D target detection model. Furthermore, in the high-performance multi-stage 3D target detection model based on hierarchical refinement set up by the scheme of the present application, based on the hierarchical refinement module and the dynamic sample allocation module, a multi-stage refinement network of an efficient multi-stage 3D target detection algorithm, namely HRNet (Hierarchical Refinement Network) network, is proposed. The use of this multi-stage refinement network for 3D target detection can at least solve the problem that the 3D target detection model based on point cloud is difficult to optimize the prediction of target category and position simultaneously, thereby affecting the accuracy of 3D target detection, and is conducive to improving the accuracy of 3D target detection.

[0137] Specifically, in the scheme of the present application, the HRNet network structure includes an RPN network module, an HRN network module and an SAP module. The RPN network module is a region proposal network (Region Proposal Network, RPN) module. The HRN network module is a hierarchical refinement network (Hierarchical Refinement Network) module. The SAP module is a scale-aware attentive propagation (Scale-aware Attentive propagation, SAP) module.

[0138] The RPN network module is used to generate region proposals in each of two or more stages based on the point cloud data training set of the target to be detected.

[0139] The HRN network module is configured to perform hierarchical refinement network and dynamic sample allocation processing on the region proposal of each of the two or more stages to obtain a hierarchical refinement result. Specifically, the HRN network module comprises a hierarchical refinement module and a dynamic sample selection module (DSS module). The hierarchical refinement module is configured to select samples of each stage, and perform hierarchical refinement on the region proposal based on multi-scale voxel features to obtain a hierarchical refinement result. The dynamic sample selection module is configured to dynamically select true positive samples of the corresponding stage for regression in the sample selection process of each stage.

[0140] The SAP module is configured to train based on the hierarchical refinement result, and propagate the obtained training result between different scale features to train the point cloud data training set of the to-be-detected target in a hierarchical refinement and dynamic sample allocation manner to obtain the 3D target detection model of the to-be-detected target.

[0141] In some embodiments, the control unit 104 trains the point cloud data training set of the to-be-detected target in a hierarchical refinement and dynamic sample allocation manner based on the point cloud data training set of the to-be-detected target by using an HRNet network structure to obtain the 3D target detection model of the to-be-detected target, including:

[0142] The control unit 104 is specifically further configured to perform one or more of point cloud preprocessing, data enhancement, object definition, point cloud voxelization, voxel encoding, and 3D feature extraction and 2D feature extraction based on the point cloud data training set of the to-be-detected target to obtain the region proposal of each of the two or more stages. The specific functions and processes of the control unit 104 are also described in step S210.

[0143] The control unit 104 is specifically further configured to select samples of each stage, and perform hierarchical refinement network processing on the region proposal based on multi-scale voxel features to obtain a hierarchical refinement result. In the sample selection process of each stage, true positive samples of the corresponding stage are dynamically selected for regression to obtain selected samples of each stage. The specific functions and processes of the control unit 104 are also described in step S220.

[0144] The control unit 104 is specifically further configured to continue training based on the hierarchical refinement result, and continue to refine the training result of each layer using features of multiple scales to finally obtain the 3D target detection model of the to-be-detected target. The specific functions and processes of the control unit 104 are also described in step S230.

[0145] In the scheme of the present application, a HRNet (Hierarchical Refinement Network) network with hierarchical refinement capability is proposed to achieve better detection performance. Specifically, the scheme of the present application performs hierarchical refinement on the region proposal (i.e. region proposal) on the multi-scale voxel feature to gradually effectively improve the performance of the regression branch. The scheme of the present application proposes a Scale-aware Attentive propagation (SAP) module to perform feature propagation between different scale features to utilize more spatial information for object detection. In order to improve the sample selection of each stage, the scheme of the present application proposes a Dynamic Sample Selection (DSS) module to select more true positive samples for regression.

[0146] In some embodiments, the control unit 104, for the region proposal of each of the two or more stages, selects samples of each stage, and performs hierarchical refinement network processing on the region proposal on the multi-scale voxel feature to obtain hierarchical refinement results, including: the control unit 104, specifically further configured to, for the region proposal of each of the two or more stages, perform hierarchical refinement network processing using the following formula to obtain hierarchical refinement results:

[0147] Cls k ,Reg k =f k (R k-1 (Cls k-1 ,Reg k-1 ))。

[0148] Wherein k represents the k-th layer voxel feature, R represents a candidate box encoding function for converting a regression vector into an (x, y, z, l, w, h, θ) format from a region proposal, Cls represents a classification prediction result, Reg represents a position prediction result, R represents a candidate box encoding function, f k represents a feature extraction method, i.e. a channel attention based feature propagation function.

[0149] Specifically, the scheme of the present application proposes a high-performance multi-stage 3D target detection algorithm based on hierarchical refinement, which can include the following steps 1 to step 10.

[0150] Step 1, point cloud preprocessing stage.

[0151] Define the point cloud as Wherein p i{x, y, z, r} represents a point in the point cloud, x, y, z represents the three-dimensional coordinates of the point, r represents the reflectivity of the point, i represents the index of the point in the point cloud, n represents the number of points in the point cloud, and i, n are positive integers. First, the point cloud is voxelized, represented as wherein L, W, H are the number of voxels in the x, y, z directions. In the point cloud preprocessing stage, first, the input point cloud is filtered. In order to limit the size of the point cloud sampling space, only points within the specified range (x min ,x max ), (y min ,y max ), (z min ,z max ) along the x, y, z directions of the point cloud are retained, and points outside the range are discarded. Then, foreground objects with a number of points less than a threshold t are filtered. Objects with too few reflection points are considered to contain insufficient information to express the spatial geometric information of the object, making it difficult to identify them, so these objects are not marked as real values.

[0152] Step 2, data augmentation, which can include the following steps 21 to 24.

[0153] Step 21, increase the detected target: store all scene real values in a dictionary, and randomly select real values from the dictionary to add to the current point cloud during training, which increases the number of targets in each frame of point cloud and simulates objects in different environments. When adding, a physical collision test is performed to remove sampling objects that collide with other objects.

[0154] Step 22, global rotation and scaling: randomly scale all points in the point cloud by a factor in the range [0.95, 1.05] and rotate by a random angle in the range .

[0155] Step 23, real value rotation and linear transformation: perform a random linear transformation subject to a Gaussian distribution on all real boxes and points within the real boxes, and rotate by a random angle in the range to simulate real values of different sizes and turns.

[0156] Step 24, flip: randomly flip the point cloud along the x-axis.

[0157] Step 3, Anchor definition (i.e., object definition).

[0158] ​The scheme of the present application trains three categories of cars, pedestrians and bicycles uniformly, wherein the car anchor frame size is set to 3.90m, 1.60m and 1.56m in x, y and z directions respectively, the pedestrian anchor frame size is set to 0.80m, 0.60m and 1.73m in x, y and z directions respectively, and the bicycle anchor frame size is set to 1.76m, 0.60m and 1.73m in x, y and z directions respectively. For the positive and negative sample division of the car category, if the IoU between the anchor frame predicted by the model and the true value is greater than 0.60, it is divided into a positive sample, and if it is less than 0.45, it is divided into a negative sample, and the predicted anchor frame between 0.45 and 0.60 is discarded. For pedestrians and bicycles, if the IoU between the anchor frame predicted by the model and the true value is greater than 0.50, it is divided into a positive sample, and if it is less than 0.35, it is divided into a negative sample, and the predicted anchor frame between 0.35 and 0.50 is discarded.

[0159] Step 4, point cloud voxelization stage.

[0160] The point cloud has the characteristics of disorder, irregularity and permutation invariance, and cannot be used for feature extraction by convolution operation in the related scheme. The voxel method first divides the irregular point cloud space into uniform stacked voxels, and then the model processes the point cloud with voxels as the minimum unit, and extracts features from the point cloud by 3D convolution, which has higher efficiency compared with directly processing points. First, the point cloud is equally divided, and the size of each voxel in the x, y and z directions is defined as V x , V y , V z , respectively, and the number of voxel grids in the x, y and z directions is , respectively, wherein D, W and H are the number of voxel grids in the depth, width and height directions, respectively. In the training stage, the number of voxel grids is limited to a maximum of 16,000.

[0161] Given the original point cloud p n =(x n ,y n ,z n ,r n ), wherein x n , y n , z n are the spatial coordinates of the point, and r n is the reflectivity information of the point. The filtered points are placed in the corresponding voxel grid, and the point p n corresponding to the voxel grid coordinates is

[0162] For data alignment, the point cloud has a value range of [0.0, 70.4] m in the x-axis direction, a value range of [-40.0, 40.0] m in the y-axis direction, and a value range of [-3.0, 1.0] m in the z-axis direction. The voxel block size in the x, y, and z directions is 0.05 m, 0.05 m, and 0.10 m, respectively, and the number of voxel blocks in the three directions is 1 408, 1 600, and 40, respectively. The number of points in each voxel is set to not more than 5.

[0163] Step 5, voxel encoding stage.

[0164] First, define the non-empty voxel where i = 1, …, n. For each voxel V, the average coordinate value of the points contained therein is calculated as the centroid (v x y z The distance from the point to the centroid is used to enhance the characteristics of each point, and the reflectivity r i is added as supplementary information, and where V in is the voxel feature, i = 1, …, n. Then, V in is converted to an m-dimensional feature space by PointNet, denoted as f Then, the maximum pooling function is used to aggregate f i to obtain the feature of the voxel V

[0165] Step 6, 3D feature extraction network.

[0166] 3D sparse convolution and sub-manifold convolution are used for multi-scale feature extraction of voxels, and convolution operation is only performed on non-empty voxels. Down-sampling is performed by 1x, 2x, 4x, and 8x, respectively, and the last layer of features is compressed on a two-dimensional plane to generate region proposals.

[0167] Step 7, 2D feature extraction network.

[0168] The 2D backbone network uses a convolutional neural network with a feature pyramid structure for multi-scale feature extraction, containing two channels: the first channel maintains the feature map resolution size unchanged and performs channel transformation in the feature dimension. The second channel maintains the feature dimension unchanged and reduces the feature map resolution size by half for feature extraction. Finally, the features of the two channels are spliced and input into the detection head to generate region proposals.

[0169] Step 8, hierarchical refinement network.

[0170] ​​Most two-stage detectors in related solutions refine the candidate boxes through fine-grained features. The candidate boxes are first sorted by classification scores, and then non-maximum suppression is performed through an IoU threshold. Compared with the classification branch, the regression branch needs to predict more variables, and the classification branch can usually approach the target true label faster. Inspired by Cascaded RCNN, in two-dimensional detection, the applicant finds that performing multi-stage refinement can improve performance. However, the computational cost is unbearable, which makes the multi-stage method difficult to be further applied in practice. Therefore, the scheme of the present application introduces a hierarchical refinement module to perform multi-stage refinement of the candidate box with higher efficiency.

[0171] In the scheme of the present application, hierarchical refinement is to optimize the position of the candidate in a top-down manner on multi-scale features, thereby continuously improving the accuracy of the candidate box. And feature propagation between different scale features is performed through an attention mechanism to improve the confidence of the candidate box. In this way, the combination of optimizing the position of the candidate and feature propagation between different scale features through an attention mechanism can improve the positioning accuracy and confidence at the same time, thereby significantly improving the detection accuracy. In each stage of target detection, the single-stage has the fastest speed but the lowest accuracy, the two-stage is slightly slower than the single-stage but has higher accuracy, and the multi-stage has the highest accuracy but the slowest speed. The scheme of the present application involves a multi-stage category of target detection, and uses a hierarchical refinement scheme to achieve the same or even faster speed as the two-stage, while having very high accuracy.

[0172] The scheme of the present application proposes a hierarchical refinement network to gradually refine the proposals generated by SECOND. Specifically, region proposals are generated at each scale of the voxel layer, and new region proposals are further refined by higher resolution voxel features. Then, the IoU threshold is dynamically recalculated at each stage to obtain more true positive samples. It is worth mentioning that the hierarchical refinement network is more efficient without increasing too much cost. It essentially uses higher scale object position information and more rich semantic and background information to guide the positioning of the candidate box, thereby propagating the object position information from lower resolution to higher resolution to obtain higher precision.

[0173] Hierarchical refinement is based on multi-scale voxel features of a three-dimensional backbone. Figure 7 is a schematic diagram of the HRNet network structure. As shown in Figure 7 each scale voxel feature is used to refine the region proposal. The high-performance multi-stage 3D target detection model based on hierarchical refinement proposed in the scheme of the present application has a total of 4 region proposal modules.

[0174] In Figure 7In the example shown, the Region Proposal Network (RPN) is used to generate candidate regions, and the hierarchical refinement module, or Hierarchical Refinement Network, is used to implement hierarchical refinement. The hierarchical refinement module can be represented as:

[0175] Cls k ,Reg k =f k (R k-1 (Cls k-1 ,Reg k-1 )).

[0176] Where k represents the voxel feature of the k-th layer, R represents the candidate box encoding function, used to convert the regression vector from the region proposal into the (x,y,z,l,w,h,θ) format, and f n This refers to a feature extraction method. In the scheme of this invention, f n This represents the Scale-aware Attentive Propagation (SAP) module. As we mentioned earlier, such as... Figure 2 As shown in (b), iterative refinement can gradually reduce classification scores. This is due to the loss of contextual information at different stages. Therefore, the present invention proposes an SAP module that aggregates contextual information progressively from top to bottom to fully utilize features at each scale. Each regressor receives features and region proposals from the previous layer as input for further refinement.

[0177] Inspired by CT3D (a two-stage 3D object detection framework), the applicant observed that the channel-wise Transformer can compute attention between different channels, avoid aggregating similar or redundant information, and integrate contextual information. Therefore, in this invention, a channel-wise Transformer is employed at each stage to obtain fine-grained feature representations for feature propagation. Specifically, in this invention, features at each scale are encoded, and then parameter-free skip connections are established between layers to enrich the features. New region proposals are propagated via the SAP module, such as... Figure 7 As shown, the formula in the SAP module is as follows:

[0178] X m =A(T) m (X m-1 V m )+X m-1 ).

[0179] wherein, X m represents the output of the mth SAP module, A represents an activation function, T m represents a channel-wise Transformer, used to fuse the context information between different channels, V m represents the mth layer of voxel features. In the scheme of the present application, there are no shared parameters in the refinement process at each stage, which makes the learning of different scale features more flexible.

[0180] Unlike image-based object detection, most of the spatial voxels of the point cloud are sparse and do not contain any points. Low-scale high-resolution voxels contain more empty voxels than high-scale low-resolution voxels. In order to gradually optimize the positioning, the refinement process starts from high-scale features, and at each refinement stage we expand each region proposal to expand the receptive field and capture more semantic and contextual information.

[0181] In some embodiments, the control unit 104 dynamically selects the true positive samples of the corresponding stage for regression in the sample selection process at each stage, to obtain the selected samples of each stage, including:

[0182] The control unit 104 is specifically further configured to find the k closest region proposals to each ground truth box in the candidate boxes in the region proposals of each stage of the two or more stages, wherein k is a hyperparameter. The specific functions and processes of the control unit 104 are also described in step S310.

[0183] The control unit 104 is specifically further configured to calculate the 3D IoU between the k candidate boxes in the k closest region proposals and the ground truth box, to obtain the 3D IoU statistical features of each ground truth box. The specific functions and processes of the control unit 104 are also described in step S320.

[0184] The control unit 104 is specifically further configured to calculate the average and standard deviation of the 3D IoU of each ground truth box and the candidate box in each frame of point cloud based on the 3D IoU statistical features of each ground truth box, as the final IoU threshold to divide the positive and negative samples, to obtain the selected samples of each stage. Wherein, in the final IoU threshold to divide the positive and negative samples, when the candidate box is closer to the ground truth box, the threshold is dynamically changed according to the distribution of IoU, and the candidate box with IoU greater than the set threshold is selected as the positive sample for regression. The specific functions and processes of the control unit 104 are also described in step S330.

[0185] Further, the high-performance multi-stage 3D object detection algorithm based on hierarchical refinement proposed in the scheme of the present application can further include the following steps 9 to step 10.

[0186] Step 9, dynamic sample allocation.

[0187] During training, only positive samples (i.e. positive samples) are used for the calculation of the regression loss, and a fixed IoU threshold is used in the related scheme to divide positive and negative samples. When hierarchical refinement is performed, the candidate frame is closer to the ground truth at each stage, which causes the distribution of IoU to change, and a fixed IoU threshold cannot effectively select samples. In order to adapt to the change of IoU distribution caused by inaccurate network parameters, different densities of voxel centers, and different distances between objects and sensors, the scheme of the present application proposes a dynamic sample selection (DSS) module, which can use the statistical characteristics of IoU to adaptively select 3D samples. In this way, the high-performance multi-stage 3D target detection model based on hierarchical refinement can recalculate the IoU threshold at each refinement stage to select different positive and negative samples, so that the selection of positive and negative samples is simple and effective, and does not bring additional calculation.

[0188] Dynamic sample allocation is because when training the model, the target position needs to be detected, some frames (i.e. candidate frames) are set in advance, which need to be divided into positive samples (i.e. samples with high coincidence degree with the target) and negative samples (i.e. samples with low coincidence degree with the target), and then the positive samples are used to calculate the regression loss of the model (to solve the positioning problem of the frame). The method of dividing positive and negative samples is that the related scheme is based on a fixed threshold (for example, 0.5) to divide positive and negative samples, but for a multi-stage target detection method, each stage will produce more accurate frames (i.e. candidate frames), and in theory this threshold should be increased as the frame position becomes more accurate, so if a fixed threshold is used, the quality of the positive samples will be reduced. In order to obtain frames with higher quality, the scheme of the present application uses the calculation method of the mean and variance to determine the threshold for sample division. Specifically, the algorithm generates multiple candidate frames on each target, then calculates the coincidence degree (i.e. intersection over union) of these candidate frames and the target, calculates the mean and variance of the coincidence degrees of all candidate frames, and uses the mean and variance as the threshold for dividing positive and negative samples. In this way, positive and negative samples can be allocated according to different thresholds at each stage, so that the detection accuracy can be obviously improved.

[0189] In general, the number of proposals is usually much larger than the ground truth, so the scheme of the present application finds the k regions closest to the real frame G k , where k is a hyperparameter. The distance d is calculated from the center point of the region proposal and the real frame, and the calculation formula is:

[0190]

[0191] where x, y, z represent the center of the box, subscript p represents the region proposal, and g represents the ground truth box.

[0192] Before performing non-maximum suppression, each ground truth box corresponds to multiple candidate boxes. In the scheme of the present application, the 3D IoU between the nearest k candidate boxes and the ground truth box is first calculated to obtain the 3D IoU statistical feature of each ground truth box. Then, the average and standard deviation of the 3D IoU of each ground truth box and the candidate box in each frame of point cloud are calculated, that is, I m and I s The DSS module can learn a specific IoU threshold for each ground truth box. Then, I = I m + I s is taken as the final IoU threshold to divide the positive and negative samples. When the candidate box is closer to the ground truth box, the threshold can be dynamically changed according to the distribution of IoU. Finally, the candidate box with IoU greater than I is selected as the positive sample for regression.

[0193] The purpose of introducing the IoU statistical feature for selecting samples is that if the quality of the candidate box is very high, then the IoU of most candidate boxes is greater than the average I m . This is unreasonable for selecting candidate boxes with different quality of IoU. For those containing objects with sparse points, the quality of the candidate box is usually low, and the IoU is also uneven, and a larger IoU threshold makes it difficult for the model to focus on these difficult samples. Therefore, I m is used to adjust the candidate boxes with different quality of IoU. In addition, the candidate boxes of objects with sparse points often have great differences in position. I s is used to increase the threshold to improve the quality of the positive samples. This makes the sample selection for objects with different sparsity more balanced.

[0194] Step 10, training loss.

[0195] In the scheme of the present application, the loss of the HRNet network is a multi-task loss composed of the RPN loss L rpn and the hierarchical refinement loss The network is trained in an end-to-end manner without any pre-training operation. The calculation method of the total loss L total of the HRNet network is as follows:

[0196]

[0197] where i represents the i-th hierarchical refinement stage. Specifically, the HRNet contains 3 refinement stages.

[0198] The RPN detection head is composed of two fully connected layers containing a nonlinear activation function. The RPN loss L rpn contains a confidence prediction loss Lcls and regression loss L reg , can be expressed as:

[0199]

[0200] where c g and r g represent the classification label and regression target, and respectively represent the predicted value. L cls The FocalLoss loss function is used to balance the contribution of positive and negative samples to the loss:

[0201]

[0202] where α and γ are the hyperparameters of Focal Loss, which are set to 0.25 and 2 respectively in the training stage.

[0203] L reg The Smooth-L1 loss function is used to calculate the loss of only positive samples:

[0204]

[0205] where p represents the number of positive samples. The total L rpn loss is calculated as follows:

[0206]

[0207] The calculation of L rpn is similar to L , which is composed of classification loss and regression loss . The cross-entropy loss function is used, and the calculation formula is:

[0208]

[0209] where, represents the confidence score of the i-th stage, and the cross-entropy loss function is used in the scheme of the present application to calculate Then add the loss of all stages to get the final classification loss. The regression loss of each stage is summed up, and the regression loss of each stage is L rpn . reg The L

[0210] The effectiveness of the proposed model is verified using the KITTI dataset, and the official evaluation indicators of KITTI are used for testing in the test set. The accuracy is calculated using 40 recall positions. Figure 8This table compares HRNet with state-of-the-art vehicle detection methods. Experimental results on the KITTI test set are shown below. Figure 8 The comparison table shown compares HRNet with state-of-the-art car detection methods, with accuracy calculated using 40 recall locations. Extensive experiments on KITTI demonstrate that the proposed high-performance multi-stage 3D object detection model based on hierarchical refinement and its corresponding algorithm achieve high detection accuracy, particularly for car detection, reaching an advanced level. KITTI is one of the most important test sets in the field of autonomous driving, primarily targeting image processing techniques for autonomous driving, mainly applied to perception and prediction. Experimental results show that the scale-aware attention propagation module (such as the hierarchical refinement module) and dynamic sample selection module proposed in this invention significantly improve detection performance. The HRNet network in this invention outperforms other advanced detectors with significantly higher performance.

[0211] Figure 9 This is a visualization of the detection effect, where (a) shows the effect. Figure 1 (b) represents the effect. Figure 2 (c) represents the effect. Figure 3 The detection visualization effect of the solution of the present invention is as follows: Figure 9 As shown.

[0212] Figure 10 The diagrams illustrate the optimization effects of the detection box at each stage, where (a) shows the optimization effect of the detection box in the second stage, (b) shows the optimization effect of the detection box in the second stage, and (c) shows the optimization effect of the detection box in the second stage. The optimization effects of each stage of the present invention on the detection effect are as follows: Figure 10 As shown.

[0213] Since the processing and functions implemented by the device in this embodiment are basically the same as the embodiments, principles and examples of the aforementioned methods, any details not covered in the description of this embodiment can be found in the relevant descriptions in the aforementioned embodiments, and will not be repeated here.

[0214] The technical scheme of the present application is characterized in that a high-performance multi-stage 3D target detection model based on hierarchical refinement is arranged, the high-performance multi-stage 3D target detection model based on hierarchical refinement comprises a hierarchical refinement module and a dynamic sample distribution module, wherein the hierarchical refinement module can use multi-scale voxel features to optimize a regression branch successively, and finally realize a higher positioning accuracy; the dynamic sample distribution module can adaptively learn an intersection over union threshold value of positive and negative sample division in each stage, to improve the accuracy of sample division; further, based on the hierarchical refinement module and the dynamic sample distribution module, a multi-stage refinement network of an efficient multi-stage 3D target detection algorithm, i.e., an HRNet network, is arranged, and 3D target detection is realized by using the multi-stage refinement network, which can obviously improve the detection accuracy.

[0215] According to the embodiment of the present application, a terminal corresponding to the efficient 3D target detection device is also provided. The terminal can comprise the efficient 3D target detection device described above.

[0216] Since the processing and functions realized by the terminal of the present embodiment are basically corresponding to the embodiments, principles and examples of the foregoing device, the description of the present embodiment will not be elaborated in detail, and the related descriptions in the foregoing embodiments can be referred to, which will not be repeated here.

[0217] The technical scheme of the present application is characterized in that a high-performance multi-stage 3D target detection model based on hierarchical refinement is arranged, the high-performance multi-stage 3D target detection model based on hierarchical refinement comprises a hierarchical refinement module and a dynamic sample distribution module, wherein the hierarchical refinement module can use multi-scale voxel features to optimize a regression branch successively, and finally realize a higher positioning accuracy; the dynamic sample distribution module can adaptively learn an intersection over union threshold value of positive and negative sample division in each stage, to improve the accuracy of sample division; further, based on the hierarchical refinement module and the dynamic sample distribution module, a multi-stage refinement network of an efficient multi-stage 3D target detection algorithm, i.e., an HRNet network, is arranged, and 3D target detection is realized by using the multi-stage refinement network, which can obviously improve the detection accuracy.

[0218] According to the embodiment of the present application, a storage medium corresponding to the efficient 3D target detection method is also provided, the storage medium comprises a stored program, wherein when the program runs, the device where the storage medium is located is controlled to execute the efficient 3D target detection method described above.

[0219] Since the processing and functions realized by the storage medium of the present embodiment are basically corresponding to the embodiments, principles and examples of the foregoing method, the description of the present embodiment will not be elaborated in detail, and the related descriptions in the foregoing embodiments can be referred to, which will not be repeated here.

[0220] By adopting the technical scheme of the present application, the layered refinement-based high-performance multi-stage 3D target detection model is set, which comprises a layered refinement module and a dynamic sample distribution module, wherein the layered refinement module can use multi-scale voxel features to optimize the regression branch one by one, and finally realize higher positioning accuracy; the dynamic sample distribution module can adaptively learn the intersection-over-union threshold of each stage positive and negative sample division to improve the accuracy of sample division; further, based on the layered refinement module and the dynamic sample distribution module, a multi-stage refinement network of the efficient multi-stage 3D target detection algorithm, namely the HRNet network, is set, and the multi-stage refinement network is used to realize 3D target detection, which is conducive to improving the overall performance of the model, and further conducive to improving the accuracy of 3D target detection.

[0221] In summary, those skilled in the art can easily understand that the above-mentioned advantageous modes can be freely combined and superimposed without conflict.

[0222] The above only describes the embodiments of the present application and is not intended to limit the present application. Those skilled in the art can make various modifications and changes to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the scope of the claims of the present application.

Claims

1. A high-efficiency 3D target detection method, characterized in that, The method comprises the following steps: obtain a point cloud data training set of a target to be detected; based on the point cloud data training set of the target to be detected, use an HRNet network structure to train the point cloud data training set of the target to be detected in a hierarchical refinement and dynamic sample allocation manner, and obtain a 3D target detection model of the target to be detected, comprising: based on the point cloud data training set of the target to be detected, perform one or more of the following processes: point cloud preprocessing, data enhancement, object definition, point cloud voxelization, voxel encoding, 3D feature extraction, and 2D feature extraction, to obtain a region proposal of each stage in two or more stages; for the region proposal of each stage in the two or more stages, select samples of each stage, and perform hierarchical refinement network processing on the region proposal on multi-scale voxel features to obtain a hierarchical refinement result, comprising: use the following formula to perform hierarchical refinement network processing to obtain a hierarchical refinement result: ， wherein, K represents the first K layer voxel feature, R represents a candidate box encoding function for converting a regression vector into format, Cls represents a classification prediction result, Reg represents a position prediction result, R represents a candidate box encoding function, f k represents a feature extraction method, i.e., a scale-aware attention propagation (SAP) module, and a formula of the SAP module is as follows: , wherein, X m represents the output of the mth SAP module, A represents an activation function, T m represents a channel-wise Transformer for fusing context information across different channels, V m represents the mth layer of voxel features; In the sample selection process in each stage, the true positive samples of the corresponding stage are dynamically selected for regression to obtain selected samples of each stage; based on the hierarchical refinement result, continue training, and continue to refine the training result of each layer using features of multiple scales to finally obtain the 3D target detection model of the target to be detected; obtain a point cloud data detection set of the target to be detected; use the 3D target detection model of the target to be detected to detect the point cloud data detection set of the target to be detected, and realize 3D target detection of the target to be detected.

2. The efficient 3D object detection method of claim 1, wherein, In the sample selection process in each stage, the true positive samples of the corresponding stage are dynamically selected for regression to obtain selected samples of each stage, comprising: For each of the candidate boxes in the region proposals for each of the two or more stages, find the closest distance to each ground truth box K where K is a hyper-parameter; The closest K region proposal is calculated K 3D IoU between the candidate box and the real box, and the 3D IoU statistics of each real box are obtained based on the 3D IoU statistical features of each real box, calculate the average and standard deviation of the 3D IoU of each real box and the candidate box in each frame of point cloud as the final IoU threshold to divide the positive and negative samples, and obtain the selected samples of each stage; wherein, in the final IoU threshold to divide the positive and negative samples, when the candidate box is closer to the real box, the threshold is dynamically changed according to the distribution of IoU, and the candidate box with IoU greater than the set threshold is selected as the positive sample for regression.

3. A high-efficient 3D object detection apparatus, characterized in that, The method comprises the following steps: an obtaining unit configured to obtain a point cloud data training set of a target to be detected; a control unit configured to, based on the point cloud data training set of the target to be detected, use an HRNet network structure to train the point cloud data training set of the target to be detected in a hierarchical refinement and dynamic sample allocation manner, and obtain a 3D target detection model of the target to be detected, comprising: based on the point cloud data training set of the target to be detected, perform one or more of the following processes: point cloud preprocessing, data enhancement, object definition, point cloud voxelization, voxel encoding, 3D feature extraction, and 2D feature extraction, to obtain a region proposal of each stage in two or more stages; for the region proposal of each stage in the two or more stages, select samples of each stage, and perform hierarchical refinement network processing on the region proposal on multi-scale voxel features to obtain a hierarchical refinement result, comprising: The hierarchical refinement network is processed by using the following formula to obtain a hierarchical refinement result: , wherein, K represents the first K layer voxel feature, R represents a candidate box encoding function for converting a regression vector into format from a region proposal, Cls represents a classification prediction result, In the sample selection process in each stage, the true positive samples of the corresponding stage are dynamically selected for regression to obtain the selected samples of each stage; represents a position prediction result, R represents a candidate box encoding function, f k represents a feature extraction method, i.e., a scale-aware attention propagation (SAP) module, a formula of the SAP module is as follows: ; wherein, X m represents the output of the mth SAP module, A represents an activation function, T m represents a channel-wise Transformer for fusing context information across different channels, V m represents the mth layer of voxel features; Based on the hierarchical refinement result, a plurality of scales of features are used to continue to refine the training results of each layer to finally obtain the 3D target detection model of the target to be detected; The acquisition unit is further configured to acquire the point cloud data detection set of the target to be detected; The control unit is further configured to detect the point cloud data detection set of the target to be detected by using the 3D target detection model of the target to be detected, and realize 3D target detection of the target to be detected. In the sample selection process in each stage, the control unit dynamically selects the true positive samples of the corresponding stage for regression to obtain the selected samples of each stage, including:

4. The efficient 3D object detection apparatus of claim 3, wherein, K For each of the candidate boxes in the region proposals for each of the two or more stages, find the closest distance to each ground truth box Based on the 3D IoU statistical features of each real box, the average and standard deviation of the 3D IoU of each real box and the candidate box in each frame of point cloud are calculated as the final IoU threshold to divide the positive and negative samples, and the selected samples of each stage are obtained; wherein, in the final IoU threshold to divide the positive and negative samples, when the candidate box is closer to the real box, the threshold is dynamically changed according to the distribution of IoU, and the candidate box with IoU greater than the set threshold is selected as the positive sample for regression. where K is a hyper-parameter; The closest K region proposal is calculated K 3D IoU between the candidate box and the real box, obtaining the 3D IoU statistical features of each real box; Including:

5. A terminal, characterized by comprising: The efficient 3D target detection device according to claim 3 or 4. The storage medium includes a stored program, wherein when the program runs, the device where the storage medium is located performs the efficient 3D target detection method according to claim 1 or 2.

6. A storage medium, characterized by ​

Images