3D target detection method and system based on superpoint

Abstract

The invention relates to a super-point-based 3D target detection method and system, and the method comprises the steps: carrying out the cluster division of an input 3D disordered point cloud, generating M point cloud blocks, carrying out the clustering of the M point cloud blocks through a k-means algorithm, so as to generate M super-points, carrying out the feature fusion of the M super-points through a feature detection algorithm and a self-attention mechanism, and carrying out the feature fusion of the M super-points. According to the method, M super points are mapped into a high-latitude semantic vector, point compression and dimension improvement are performed on the high-latitude semantic vector through multi-round feature fusion, the high-latitude semantic vector is converted into a pseudo image, multi-scale feature extraction is performed through an FPN network, a target feature map is generated, and the target feature map is decoupled through a decoupling detection head. And generating a decoupling result including confidence, calculating a loss value of the confidence through a loss function, filtering the decoupling result lower than a loss threshold, and generating a 3D target detection result according to the filtered decoupling result. Therefore, 3D target detection is realized in a more efficient manner, and the detection precision is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a 3D target detection method and system based on super point. BACKGROUND

[0002] In the research of 3D target detection, most of the processing methods based on point cloud rely on the spatial voxel modeling strategy. This kind of method usually divides the 3D point cloud into fixed size voxels or cylinders uniformly, extracts feature embedding representation from them, and projects them onto the bird's eye view compatible with the standard 2D convolution to adapt to the local feature extraction of the subsequent backbone network. Among them, the cylinder is the extension of the voxel, which performs better in encoding speed and backbone network processing efficiency. However, whether it is voxel or cylinder, it is based on the spatial position in the point cloud world. Since the point cloud distribution is sparse and irregular, the point cloud in each voxel or cylinder is often irregularly distributed, resulting in a huge difference in point cloud in different voxels or cylinders, some of which are less, some of which are more, and some of which even do not exist. In order to unify the number of point clouds in each voxel or cylinder, the existing method usually adopts the padding operation, but this operation not only introduces a large amount of redundant calculation and memory occupation without actual semantics, but also causes significant space waste, affecting the subsequent detection accuracy.

[0003] From the detection paradigm, most of the 3D target detection methods adopt the anchor box based detection mechanism. This method allocates an anchor box for each target in the 3D world and trains and reasons according to the anchor box. Although this mode can improve the detection accuracy to a certain extent, since the objects in the 3D point cloud world not only have great size difference, but also have diversity in orientation, the anchor box cannot exhaust all object orientations. If the orientation difference is forcibly covered by increasing the number of anchor boxes, the total amount of anchor boxes will be inflated sharply, further aggravating the imbalance problem of positive and negative samples, and the proportion of a large number of invalid anchor boxes as negative samples and a small number of effective positive samples is seriously out of balance, increasing the difficulty of model training and the complexity of reasoning, and affecting the subsequent detection accuracy. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a 3D target detection method and system based on super point to realize 3D target detection in a more efficient way and improve detection accuracy.

[0005] In order to solve the above technical problems, the technical scheme adopted by the present application is: In a first aspect, the present application provides a 3D target detection method based on super point, comprising: obtaining an input 3D unordered point cloud, clustering the 3D unordered point cloud to generate M point cloud blocks, and generating M super points by clustering the M point cloud blocks through a k-means algorithm; The M super points are mapped into high latitude semantic vectors by feature detection algorithm and self-attention mechanism, and the high latitude semantic vectors are compressed in points and improved in dimensions through multiple rounds of feature fusion, so as to convert the high latitude semantic vectors into pseudo images; The pseudo images are extracted in multiple scales by the FPN network to generate a target feature map containing context information, the target feature map is decoupled by a decoupling detection head to generate a decoupling result including a confidence, a loss value of the confidence is calculated by a loss function, the decoupling result with the loss value lower than a loss threshold is filtered to obtain a filtered decoupling result, and a 3D target detection result is generated according to the filtered decoupling result.

[0006] The 3D unordered point cloud is divided into clusters, and the M point cloud blocks obtained by the cluster division are clustered by the k-means algorithm to generate M super points, so that the related points of the unordered point cloud are aggregated in a clustered manner to form super points that can adapt to the sparse distribution of the 3D unordered point cloud, highlight the texture features of the point cloud, and simplify the point cloud processing procedure while avoiding redundant calculation and space waste caused by the filling operation and sequence interception, thereby improving the point cloud processing efficiency.

[0007] Optionally, the cluster division of the 3D unordered point cloud to generate M point cloud blocks and the clustering of the M point cloud blocks by the k-means algorithm to generate M super points include: The 3D unordered point cloud is sorted in ascending order according to the x-axis coordinates to obtain an ascendingly sorted 3D unordered point cloud, and the ascendingly sorted 3D unordered point cloud is divided along the positive direction of the x-axis at a preset fixed interval to generate M point cloud blocks that can be processed in parallel. Each point cloud block is sampled by FPS to obtain M initial clustering centers, and the M initial clustering centers are clustered by the k-means algorithm, and the clustering is stopped when the number of clustered points reaches n, so as to generate M super points containing n points.

[0008] According to the above description, the 3D unordered point cloud is divided into M point cloud blocks that can be processed in parallel, which greatly reduces the computational density of subsequent FPS sampling and clustering, improves processing efficiency and scalability, performs FPS sampling on each point cloud block to ensure that the obtained initial cluster centers are widely distributed and cover the entire geometric space, effectively avoids the problem of uneven distribution of cluster centers caused by random initialization, and ensures the quality and feature consistency of subsequent clustering. The size of the super point is strictly controlled to eliminate computational redundancy.

[0009] Optionally, the generating M super points each containing n points includes: calculating the center point of each super point, and sorting the M super points each containing n points in ascending order according to the y coordinates of the center points to obtain the M super points each containing n points sorted in ascending order.

[0010] According to the above description, the M super points each containing n points are calculated and sorted in ascending order according to the y coordinates of the center points to maintain the original relative position relationship between the super points, i.e., to maintain the spatial structure information.

[0011] Optionally, the feature fusion of the M super points through the feature detection algorithm and the self-attention mechanism to map the M super points to high-dimensional semantic vectors includes: extracting features of the M super points respectively through the feature detection algorithm to obtain initial features of the M super points; aggregating query vectors of the M super points to construct a global query layer, constructing a multi-head single-query attention layer based on the global query layer and the self-attention mechanism, converting the initial features to high semantic features through the attention layer, and concatenating high semantic features of every k adjacent super points to generate high-dimensional high semantic aggregation points, thereby mapping the M super points to high-dimensional semantic vectors.

[0012] According to the above description, the global query layer aggregated from all query vectors is combined with the self-attention mechanism to construct a multi-head single-query attention layer, which reduces the computational complexity compared with multi-head multi-query attention, thereby learning the context information in the super points with low spatial overhead and computational complexity, converting the initial features to high semantic features, concatenating high semantic features of every k adjacent super points to generate high-dimensional high semantic aggregation points, which not only compresses the super points, but also makes the feature expression more concentrated, and improves the ability of the model to capture small-scale target features.

[0013] Optionally, the obtaining of the initial features of the M super points includes: The center point of each hyperpoint is calculated according to all points in each hyperpoint, and the offset vector of the point in each hyperpoint and the center point of the corresponding hyperpoint is calculated, and the included angle formed by the point in each hyperpoint, the corresponding offset vector and the origin is calculated by a first formula, and the first formula is: ; wherein, represents the included angle formed by the point i of the hyperpoint, the corresponding offset vector and the origin, represents the x-axis coordinate of the point i of the hyperpoint, represents the y-axis coordinate of the point i of the hyperpoint, represents the z-axis coordinate of the point i of the hyperpoint, represents the x-axis coordinate of the center point of the hyperpoint, represents the y-axis coordinate of the center point of the hyperpoint, represents the z-axis coordinate of the center point of the hyperpoint. A second formula of the nearest neighbor method is used to calculate the normal vector of each hyperpoint, the initial feature and the corresponding offset vector, the corresponding included angle and the corresponding normal vector are spliced to realize feature enhancement of the initial feature, and the initial feature after feature enhancement is obtained, and the second formula is: ; wherein, represents the normal vector, and n represents the number of points of the hyperpoint, represents the x-axis coordinate of the point i of the hyperpoint, represents the y-axis coordinate of the point i of the hyperpoint, represents the z-axis coordinate of the point i of the hyperpoint, represents the x-axis coordinate of the point j of the hyperpoint, represents the y-axis coordinate of the point j of the hyperpoint, represents the z-axis coordinate of the point j of the hyperpoint.

[0014] According to the above description, the offset vector is the relative position of the point in the hyperpoint and the center point of the corresponding hyperpoint, which helps the model to perceive the subtle distribution mode of the point in the hyperpoint, the normal vector provides the orientation information of the local surface where the hyperpoint is located, and the included angle provides the context information of the point in the hyperpoint, so that the initial feature and the offset vector, the included angle and the normal vector are spliced, which greatly enriches the representation information of the hyperpoint.

[0015] Optionally, the pseudo image is an XxYxD pseudo image, wherein X represents the number of hyperpoints on the x-axis, Y represents the number of hyperpoints on the y-axis, and D represents the number of channels, and D=cxn, c represents the channel length of the point in the hyperpoint, and n represents the number of points of the hyperpoint.

[0016] According to the above description, the number of channels is composed of the number of superpoints and the channel length of the points in the superpoint, retains and encapsulates rich point-level information, avoids information loss caused by early aggregation, and balances the spatial structure and feature density.

[0017] Optionally, the multi-scale feature extraction of the pseudo image by the FPN network to generate a target feature map containing context information comprises: The FPN network comprises a first connection unit from bottom to top composed of a plurality of cascaded ResUnit modules, a second connection unit from top to bottom composed of a plurality of cascaded CSPUnit modules, and a third connection unit connecting the first connection unit and the second connection unit composed of a DenseNet; The first connection unit is used for multi-scale feature extraction of the pseudo image by downsampling to generate a base feature map, the second connection unit is used for multi-scale feature extraction of the base feature map by upsampling to generate a reference feature map, and the third connection unit is used for feature fusion of the base feature map and the reference feature map to generate a target feature map containing context information. The ResUnit module adopts a Swish function as an output activation function.

[0018] According to the above description, the first connection unit is used for multi-scale feature extraction of the pseudo image by downsampling, i.e., multi-scale feature extraction by a path from bottom to top to generate a base feature map containing rich detail information, the residual structure of the ResUnit module can alleviate the gradient vanishing problem of the deep network, improve the stability of training, and the ResUnit module adopts a Swish function as an output activation function to further improve the smoothness of feature expression and gradient flow, the second connection unit is used for multi-scale feature extraction of the base feature map by upsampling, i.e., multi-scale feature extraction by a path from top to bottom to restore the spatial resolution to generate a reference feature map, enhance the diversity of gradients, and improve the learning ability, and the third connection layer is used for connecting the base feature map and the reference feature map for feature fusion to realize the fusion of bottom details and high-level semantics to generate a target feature map containing context information.

[0019] Optionally, the decoupling of the target feature map by the decoupling detection head to generate a decoupling result comprising a confidence score comprises: A LxL window is slid on the M superpoints according to a first step size, and the superpoints in the window are merged every time the window is slid to generate m window points; The center point of each window point is calculated according to all the superpoints in each window point to obtain m window centers. Calculate the Euclidean distance between each target point on the target feature map and all window points, map the target point with the window point with the minimum Euclidean distance, and obtain the target ground truth information; Decouple the target feature map by decoupling the detection head and the target ground truth information, and generate a decoupled result including a confidence.

[0020] According to the above description, the target point on the target feature map is bound to the window point with the minimum Euclidean distance by the sliding window, which makes the feature point on the target feature map bind to at most one target, and the target overlapping situation is avoided, thereby improving the accuracy of the obtained decoupled result.

[0021] Optionally, the decoupled result includes target category and target position information.

[0022] In a second aspect, the present application provides a 3D target detection system based on super points, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the 3D target detection method based on super points according to the first aspect when executing the computer program.

[0023] The technical effects of the 3D target detection system based on super points according to the second aspect are referred to the 3D target detection method based on super points according to the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 A flowchart of the 3D target detection method based on super points according to the present embodiment; Figure 2 A schematic diagram of the overall flow of the 3D target detection method based on super points according to the present embodiment; Figure 3 A first super point diagram of the M super points according to the present embodiment; Figure 4 A second super point diagram of the M super points according to the present embodiment; Figure 5 A curve diagram for predicting the precision and recall of each category on the KITTI dataset according to the present embodiment; Figure 6 A structural schematic diagram of the 3D target detection system based on super points according to the present embodiment.

[0025] LEGEND 1. A 3D target detection system based on super points; 2. A processor; 3. A memory. DETAILED DESCRIPTION

[0026] For a better understanding of the above technical solutions, the exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present application can be more clearly, thoroughly understood and the scope of the present application can be completely conveyed to those skilled in the art.

[0027] Embodiment one Please refer to Figures 1 to 5 The present application provides a 3D target detection method based on super points, comprising the steps of: S1, obtaining an input 3D unordered point cloud, performing cluster division on the 3D unordered point cloud to generate M point cloud blocks, and clustering the M point cloud blocks by a k-means algorithm to generate M super points; At this time, the step S1 of performing cluster division on the 3D unordered point cloud to generate M point cloud blocks and clustering the M point cloud blocks by a k-means algorithm to generate M super points comprises: S11, sorting the 3D unordered point cloud in ascending order according to the x-axis coordinates to obtain an ascendingly sorted 3D unordered point cloud, and dividing the ascendingly sorted 3D unordered point cloud along the positive direction of the x-axis according to a preset fixed interval to generate M point cloud blocks that can be processed in parallel; S12, performing FPS sampling on each point cloud block respectively to obtain M initial clustering centers, clustering the M initial clustering centers by a k-means algorithm, and stopping clustering when the number of clustering points reaches n to generate M super points containing n points.

[0028] In the present embodiment, as shown in Figure 2 , an input 3D unordered point cloud is obtained, wherein the 3D unordered point cloud can be represented as: points={p i =[x i ,y i ,z i ]∈R 3} i=1…N , wherein N represents the total number of the 3D unordered point cloud, p i represents a point i in the 3D unordered point cloud, x i represents the x-axis coordinate of the point i in the 3D unordered point cloud, y i represents the y-axis coordinate of the point i in the 3D unordered point cloud, and z iz-coordinate of a point i in a 3D unordered point cloud, the 3D unordered point cloud is sorted in ascending order according to the x-coordinate to obtain an ascendingly sorted 3D unordered point cloud, and the ascendingly sorted 3D unordered point cloud is divided along the positive direction of the x-axis at a preset fixed interval to generate M point cloud blocks. Each point cloud block is subjected to FPS (Farthest Point Sampling) sampling to obtain M initial clustering centers, and the M initial clustering centers are clustered through a k-means algorithm, with the number of clustered points reaching n as a termination condition, that is, clustering is stopped when the number of clustered points reaches n, to generate M clusters containing n points, that is, superpoints, and therefore each superpoint contains n points.

[0029] At this time, the step S12 of generating M superpoints containing n points includes: S121, calculating a center point of each superpoint, and sorting the M superpoints containing n points in ascending order according to the y-coordinate of the center point to obtain the M superpoints containing n points in ascending order.

[0030] In the embodiment, as shown in Figure 2 To avoid disturbing the original relative positions between superpoints, the center point of each superpoint needs to be calculated, and the M superpoints containing n points are sorted in ascending order according to the y-coordinate of the center point to obtain the M superpoints containing n points in ascending order, so as to realize the conversion of the 3D unordered point cloud into an element graph taking the superpoint as the minimum unit.

[0031] As shown in Figures 3-4 , Figure 3 is a first superpoint diagram of the M superpoints with the x-coordinate of 8, the y-coordinate of 8, and n of 1536, Figure 4 is a second superpoint diagram of the M superpoints with the x-coordinate of 32, the y-coordinate of 32, and n of 96. As the values of the x-coordinate and the y-coordinate are larger, the second superpoint diagram displayed in the form of red and blue alternation looks more like a mottling, and the mottling is more densely distributed. Since the current explanatory diagram can only be represented by gray color, the mottling is not clear enough, and if a color explanatory diagram is needed in the future, a related color explanatory diagram can be provided.

[0032] S2, feature fusion is performed on the M superpoints through a feature detection algorithm and a self-attention mechanism, M superpoints are mapped into high-dimensional semantic vectors, and the high-dimensional semantic vectors are subjected to point number compression and dimension lifting through multiple rounds of feature fusion, so as to convert the high-dimensional semantic vectors into pseudo images; In the embodiment, as shown in Figure 2As shown, since the hyperpoints are formed in a clustering-based manner, there is no strong correlation between the points in different hyperpoints, and there is no great connection between them. Instead, more attention should be paid to the connection between hyperpoints, and local feature information should be mined from the perspective of hyperpoints. Through feature detection algorithm and self-attention mechanism, the M hyperpoints are fused to realize mapping of the M hyperpoints to high-dimensional semantic vectors. Through multiple rounds of feature fusion, the high-dimensional semantic vectors are compressed and the dimensions are improved to realize conversion of the high-dimensional semantic vectors to pseudo images, wherein the pseudo images are X×Y×D pseudo images, X represents the number of hyperpoints on the x-axis, Y represents the number of hyperpoints on the y-axis, and D represents the number of channels, and D=c×n, c represents the channel length of the points in the hyperpoint, and n represents the number of points in the hyperpoint.

[0033] At this time, the step S2 of fusing the M hyperpoints through the feature detection algorithm and the self-attention mechanism to realize mapping of the M hyperpoints to high-dimensional semantic vectors comprises: S21, the initial features of the M hyperpoints are extracted through the feature detection algorithm, respectively; At this time, the step S21 of obtaining the initial features of the M hyperpoints comprises: S211, the center point of each hyperpoint is calculated according to all the points in each hyperpoint, and the offset vector of each point in each hyperpoint and the center point of the corresponding hyperpoint is calculated, and the included angle formed by each point in each hyperpoint, the corresponding offset vector and the origin is calculated through a first formula, and the first formula is: ; wherein, represents the included angle formed by the point i of the hyperpoint and the corresponding offset vector and the origin, represents the x-axis coordinate of the point i of the hyperpoint, represents the y-axis coordinate of the point i of the hyperpoint, represents the z-axis coordinate of the point i of the hyperpoint, represents the x-axis coordinate of the center point of the hyperpoint, represents the y-axis coordinate of the center point of the hyperpoint, represents the z-axis coordinate of the center point of the hyperpoint; S212, the normal vector of each hyperpoint is calculated by a second formula of the nearest neighbor method, the initial features are spliced with the corresponding offset vector, the corresponding included angle and the corresponding normal vector to realize feature enhancement of the initial features, and the initial features after feature enhancement are obtained, and the second formula is: ; wherein, represents the normal vector, and n represents the number of points in the hyperpoint, represents the x-axis coordinate of the point i of the hyperpoint, a y-axis coordinate of a point i of the hyperpoint, a z-axis coordinate of the point i of the hyperpoint, an x-axis coordinate of a point j of the hyperpoint, a y-axis coordinate of the point j of the hyperpoint, a z-axis coordinate of the point j of the hyperpoint.

[0034] In this embodiment, as shown in Figure 2 the center point of each hyperpoint is calculated according to all points in each hyperpoint, and the offset vector of the point in each hyperpoint and the center point of the corresponding hyperpoint is calculated, the included angle formed by the point in each hyperpoint, the corresponding offset vector and the origin point is calculated by the first formula, the normal vector of each hyperpoint is calculated by the second formula of the nearest neighbor method, the initial features of the M hyperpoints are extracted by the feature detection algorithm respectively, the initial features after feature enhancement are obtained by splicing the initial features of the M hyperpoints, the corresponding offset vectors, the corresponding included angles and the corresponding normal vectors, and realizing feature enhancement of the initial features.

[0035] S22, aggregating the query vectors of the M hyperpoints to construct a global query layer, constructing a multi-head single-query attention layer based on the global query layer and a self-attention mechanism, converting the initial features into high semantic features through the attention layer, and concatenating the high semantic features of every k adjacent hyperpoints to generate high-latitude high semantic aggregation points, and realizing mapping of the M hyperpoints into high-latitude semantic vectors.

[0036] In this embodiment, as shown in Figure 2 the query vectors of the M hyperpoints are aggregated to construct a global query layer, and the global query layer and a self-attention mechanism are combined to construct a multi-head single-query attention layer, the initial features are converted into high semantic features through the attention layer, and the high semantic features of every k adjacent hyperpoints are concatenated to generate high-latitude high semantic aggregation points, at this time the number of points n of the hyperpoint will drop to n / k, and the channel length of each point will rise from c to c x k, realizing mapping of the M hyperpoints into high-latitude semantic vectors.

[0037] S3, performing multi-scale feature extraction on the pseudo image through the FPN network to generate a target feature map containing context information, decoupling the target feature map through a decoupling detection head to generate a decoupling result including a confidence, calculating a loss value of the confidence through a loss function, filtering the decoupling result whose loss value is lower than a loss threshold to obtain a filtered decoupling result, and generating a 3D target detection result according to the filtered decoupling result.

[0038] In this embodiment, as shown in Figure 2As shown, the pseudo image is subjected to multi-scale feature extraction by a FPN network (Feature Pyramid Network) to generate a target feature map containing context information, and the target feature map is subjected to decoupling by a decoupling detection head, which contains three mapping heads for performing prediction tasks of confidence, target class and target position, thereby generating decoupling results containing confidence, target class and target position information. The loss value of the confidence is calculated by a loss function, wherein the true value of the confidence is a binary value. When the true value of the confidence is 1, it indicates that there is a target center point in the current cluster, which is taken as a positive sample. When the true value of the confidence is 0, it indicates that there is no target center point, which is taken as a negative sample. The loss function used is Focal Loss. When the true value of the confidence is 1, i.e. the positive sample, the loss function of the positive sample is represented as: ; wherein, represents the loss function of the positive sample, represents a first coefficient, and = 1, represents a second coefficient, and = 2, and c represents the channel length of the super point in point. When the true value of the confidence is 0, i.e. the negative sample, the loss function of the negative sample is represented as: ; wherein, represents the loss function of the negative sample, represents a first coefficient, and = 0.25, represents a second coefficient, and = 3, and c represents the channel length of the super point in point. The loss value of the confidence is obtained by adding and summing the loss function of the positive sample and the loss function of the negative sample, and the decoupling results with a loss value lower than a loss threshold are filtered to obtain filtered decoupling results, and the 3D target detection results are generated according to the filtered decoupling results.

[0039] At this time, the step S3 of performing multi-scale feature extraction on the pseudo image by the FPN network to generate a target feature map containing context information comprises; S31, the FPN network comprises a first connection unit from bottom to top composed of a plurality of cascaded ResUnit modules, a second connection unit from top to bottom composed of a plurality of cascaded CSPUnit modules, and a third connection unit connecting the first connection unit and the second connection unit composed of DenseNet. S32, performing multi-scale feature extraction on the pseudo image by the first connection unit to generate a base feature map, performing multi-scale feature extraction on the base feature map by a second connection unit to generate a reference feature map, and performing feature fusion on the base feature map and the reference feature map by a third connection unit to generate a target feature map containing context information; In the embodiment, the ResUnit module adopts a Swish function as an output activation function.

[0040] In the embodiment, the FPN network includes a first connection unit from bottom to top composed of a plurality of cascaded ResUnit modules, a second connection unit from top to bottom composed of a plurality of cascaded CSPUnit modules, and a third connection unit connecting the first connection unit and the second connection unit composed of a DenseNet. The first connection unit is used for multi-scale feature extraction on the pseudo image by down-sampling in a bottom-up path to capture features of different scales and generate a base feature map containing rich detailed information. The ResUnit module adopts a Swish function as an output activation function. The second connection unit is used for multi-scale feature extraction on the base feature map by up-sampling in a top-down path to restore spatial resolution and generate a reference feature map. The third connection unit is used for connecting the base feature map and the reference feature map to perform feature fusion, realize the fusion of bottom details and high-level semantics, and generate a target feature map containing context information.

[0041] In the embodiment, since adjacent feature points on the target feature map share most network weight parameters and most receptive fields on the original map of the 3D unordered point cloud, one target is covered by the receptive fields of multiple feature points and can be learned by multiple feature points at the same time. It is not necessary to make a feature point perceive all targets in its receptive field, because doing so does not fully exert the advantage of shared features of the convolution window. Therefore, a true value association strategy of associating one feature point with only one target is constructed, which is as follows: The decoupling result including the confidence generated by decoupling the target feature map by the decoupling detection head in step S3 includes: S33, sliding a LxL window on the M super points according to a first step size, and merging the super points in the window each time the window is slid to generate m window centers; S34, calculating the center point of each window point according to all super points in each window point to obtain m window centers; S35, calculating the Euclidean distance between each target point on the target feature map and all window points, mapping the target point to the window point with the smallest Euclidean distance, and obtaining target true value information; S36, decoupling the target feature map by decoupling the detection head and the target ground truth information, to generate a decoupling result including confidence.

[0042] As shown in Figure 2 the LxL window is slid on the M super points according to the first step size, at this time, L is 2, the first step size is 2, each sliding once will merge the 4 super points in the window, thereby generating a larger super point, i.e. a window point, to generate m window points, the center point of each window point is calculated according to all the super points in each window point, m window centers are obtained, the Euclidean distance between each target point on the target feature map and all window points is calculated, the target point is mapped with the window point with the smallest Euclidean distance, the target ground truth information is obtained, the target feature map is decoupled by decoupling the detection head and the target ground truth information, and a decoupling result including confidence is generated.

[0043] The 3D target detection method based on super points in the embodiment can be applied to multiple scenes, such as vehicle, pedestrian and cyclist target detection.

[0044] The following specific scene application: the 3D target detection method based on super points of embodiment one is tested on Waymo open dataset and KITTI dataset, and the anchor-based and anchor-free detection mechanism and the method based on voxel processing and cylinder processing are combined in pairs on Waymo open dataset and KITTI dataset, and the test results obtained under different model architectures are compared.

[0045] The Waymo dataset provides information collected from a set of sensors on an autonomous vehicle, including multiple lidars and cameras. It captures the environment of multiple major cities in the United States under different weather conditions or at different times of the day. The dataset provides 1000 sequences, each with a duration of 20s, sampled at 10Hz, so each sequence contains approximately 200 frames of point cloud. 5 frames are randomly selected from each sequence as training samples, and 1 frame is selected as test sample, totaling 5000 training samples and 1000 test samples, and all samples are FPS sampled, and the point number is unified to 98304.

[0046] Processing of the 3D target detection method based on super points: let X=Y=64, each cluster contains 24 points, the input format of the point cloud sample is 64x64x24, and the memory size occupied is 384KB.

[0047] Voxel processing: Voxel is a further subdivision based on the column, because voxel also involves division along the z-axis direction. In the experiment, the number of division along the z-axis direction is set to 4, so on the basis of the column, for each sample, we will get 4096*4=16384 voxels. At the same time, according to the idea of the column, the number of points contained in each voxel is limited to 36. Finally, the input format of the point cloud sample after voxel processing is 4096*4*36, and the memory size occupied is 2304KB.

[0048] Column processing: the point cloud sample is divided into 4096 columns according to the spatial position, and due to the sparse distribution of the point cloud, the number of points contained in each column is different. In order to reduce the information loss as much as possible, and combined with the average number of points contained in the column, the number of points in each column is limited to 72, and the column is down-sampled more than the value, and the zero is filled less than the value. Finally, the input format of the point cloud sample after column processing is 4096*72, and the memory size occupied is 1152KB.

[0049] Anchor-based detection mechanism: all vehicle real boxes in the training sample are extracted, and KNN clustering with class 4 is performed, and the parameters considered in the clustering are (w, l, h), representing the length, frame and height of the vehicle respectively. Then according to the four size parameters obtained by clustering, an anchor box is assigned to each target vehicle as a real size label for model training.

[0050] Table 1. Performance results of different model architectures on Waymo open dataset

[0051] As shown in Table 1, the 3D target detection method based on super point of the application can free the model parameters and improve the model inference speed, which benefits from the encoding mode with super point as the core and the feature extraction idea of converting the super point distribution map into a pseudo image for 2D convolution. Because the column processing and voxel processing are fixed in the encoding form, they cannot adapt to the sparse distribution of the point cloud, resulting in a large amount of redundant information, thereby increasing the model parameter overhead and slowing down the training and inference speed. The division dimension of voxel processing is 3D, which cannot be used for 2D convolution backbone network, when using 3D convolution network, the model parameter amount increases significantly, and the inference speed decreases significantly. In addition, Anchor-based is a detection idea with anchor box as the core, which increases the output dimension of the model, thereby causing the increase of the model parameter amount and the decrease of the inference speed.

[0052] KITTI dataset has 7481 samples, and the official test set has 7518 samples. 6400 samples are selected from the official training set as training data, and the remaining 1081 are used as validation data. In this process, some mislabeled records will be cleaned. Through statistics, the entire training data contains a total of 37124 vehicles, 4296 pedestrians, and 1220 cyclists. In addition, 1600 samples are selected from the KITTI official test set as the test set. Finally, a series of analysis experiments on the above data are carried out.

[0053] According to previous experience, the model often has relatively weak learning and understanding ability for small-scale targets, and it is more difficult to capture the features of such targets. Although as the target size decreases, the AP: average precision: also decreases accordingly, such as Figure 5 The area surrounded by the three curves in the figure respectively with the coordinate axes can be known that for the present application, the amplitude of this decrease is not very large, and the AP of small-scale target pedestrians or cyclists is not much different from that of large-scale target vehicles, wherein Figure 5 The cyelist in the figure represents cyclists, the pedestrian represents pedestrians, the vehicle represents vehicles, the precision represents precision, and the recall represents recall. This shows that the present application can well mine the multi-scale information of the point cloud sequence in the real world and exhibit robustness in the multi-scale context. As shown in Table 2, compared with other classic models, the mAP: average AP: of the present application is in a leading position.

[0054] Table 2. Performance results of different model architectures on KITTI dataset

[0055] In Table 2, VoxelNet represents voxel network, SECOND represents coefficient convolution 3D detector, PointRCNN represents point cloud region convolutional neural network, PointPillars represents point cloud column network, MVF represents multi-view fusion, and CenterPoint represents center point.

[0056] Embodiment two Please refer to Figure 6 The present application provides a 3D target detection system 1 based on super points, which includes a memory 3, a processor 2, and a computer program stored on the memory 3 and executable on the processor 2. The processor 2 implements the steps in embodiment one when executing the computer program.

[0057] The system / device used for implementing the method of the embodiments of the present application described above can be understood by those skilled in the art based on the method of the embodiments of the present application described above, and thus will not be described here again. The system / device used for implementing the method of the embodiments of the present application described above all belong to the scope of the present application.

[0058] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems or computer program products. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can be embodied in the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk memory, CD-ROM, optical memory, etc.) embodying computer usable program code.

[0059] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions.

[0060] It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the claims, the word "first", "second", "third" etc. does not limit the number for these elements. These words are only used to distinguish between alternative claims. The word "plurality" does not imply a limitation to just two instances of the referenced item.

[0061] Furthermore, it is noted that the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art will recognize that the embodiments or examples described herein can be practiced with a less than all the features or a greater than all the features described herein. Thus, the described features, structures, or characteristics can be combined in one or more embodiments or examples. In addition, those skilled in the art will appreciate that the various embodiments or examples described herein can be implemented in a computer system or article of manufacture using standard programming and / or engineering techniques.

[0062] While the preferred embodiments of the application have been described, those skilled in the art will recognize that many modifications and variations of this preferred embodiment are possible without departing from the spirit or scope of the present application. Thus, it is intended that the present application embrace all such modifications and variations as fall within the scope of the appended claims and their equivalents.

[0063] Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A 3D target detection method based on superpoints, characterized in that, include: The input 3D unordered point cloud is obtained, the 3D unordered point cloud is divided into clusters to generate M point cloud blocks, and the M point cloud blocks are clustered by the k-means algorithm to generate M super points. By using feature detection algorithms and self-attention mechanisms to fuse features of M superpoints, the M superpoints are mapped into high-dimensional semantic vectors. Through multiple rounds of feature fusion, the high-dimensional semantic vectors are compressed in terms of the number of points and their dimensions are increased, thereby converting the high-dimensional semantic vectors into pseudo-images. Multi-scale feature extraction is performed on the pseudo-image using an FPN network to generate a target feature map containing contextual information. The target feature map is then decoupled using a decoupling detection head to generate a decoupling result including confidence. The loss value of the confidence is calculated using a loss function. Decoupling results with loss values ​​lower than a loss threshold are filtered to obtain filtered decoupling results. A 3D target detection result is then generated based on the filtered decoupling results.

2. The 3D target detection method based on superpoints as described in claim 1, characterized in that, The step of clustering the 3D unordered point cloud to generate M point cloud blocks, and then clustering the M point cloud blocks using the k-means algorithm to generate M superpoints includes: The 3D disordered point cloud is sorted in ascending order according to the x-axis coordinate to obtain the 3D disordered point cloud after ascending order. The 3D disordered point cloud after ascending order is divided along the positive x-axis at a preset fixed interval to generate M point cloud blocks that can be processed in parallel. For each point cloud block, perform FPS sampling to obtain M initial cluster centers. Then, use the k-means algorithm to cluster the M initial cluster centers. Stop clustering when the number of cluster points reaches n to generate M superpoints containing n points.

3. The 3D target detection method based on superpoints as described in claim 2, characterized in that, The process of generating M superpoints, each containing n points, includes: Calculate the center point of each superpoint, and sort the M superpoints containing n points in ascending order according to the y-coordinate of the center point to obtain the M superpoints containing n points in ascending order.

4. The 3D target detection method based on superpoints as described in claim 1, characterized in that, The step of fusing features of M superpoints using a feature detection algorithm and a self-attention mechanism to map the M superpoints into high-dimensional semantic vectors includes: The initial features of the M superpoints are obtained by extracting features from each of the M superpoints using a feature detection algorithm. The query vectors of M superpoints are aggregated to construct a global query layer. Based on the global query layer and the self-attention mechanism, a multi-head single query attention layer is constructed. The initial features are converted into high semantic features through the attention layer, and the high semantic features of every k adjacent superpoints are concatenated to generate high-dimensional high semantic aggregation points, thereby mapping the M superpoints into high-dimensional semantic vectors.

5. The 3D target detection method based on superpoints as described in claim 4, characterized in that, The initial features for obtaining the M superpoints include: Calculate the center point of each superpoint based on all points within each superpoint, and calculate the offset vector between each point within a superpoint and its corresponding center point. Simultaneously, calculate the angle formed by each point within a superpoint, its corresponding offset vector, and the origin using the first formula: ； in, Let i represent the angle formed by the superpoint i, the corresponding offset vector, and the origin. This represents the x-axis coordinate of point i, which is a super-point. This represents the y-coordinate of point i, which is a super-point. This represents the z-axis coordinate of point i, which is a super-point. The x-axis coordinates of the center point of the super-point. This represents the y-coordinate of the center point of the super-point. This represents the z-axis coordinate of the center point of the super-point; The normal vector of each superpoint is calculated using the second formula of the nearest neighbor method. The initial feature is then concatenated with the corresponding offset vector, the corresponding angle, and the corresponding normal vector to perform feature enhancement on the initial feature, resulting in the enhanced initial feature. The second formula is as follows: ； in, Let n represent the normal vector, and n represent the number of points that superscore. This represents the x-axis coordinate of point i, which is a super-point. This represents the y-coordinate of point i, which is a super-point. This represents the z-axis coordinate of point i, which is a super-point. This represents the x-axis coordinate of point j, which is a super-point. This represents the y-coordinate of point j, which is a super-point. This represents the z-axis coordinate of point j, which is a super-point.

6. The 3D target detection method based on superpoints as described in claim 1, characterized in that, The pseudo-image is an X×Y×D pseudo-image, where X represents the number of superpoints on the x-axis, Y represents the number of superpoints on the y-axis, D represents the number of channels, and D=c×n, where c represents the channel length of the superpoints and n represents the number of superpoints.

7. The 3D target detection method based on superpoints as described in claim 1, characterized in that, The step of extracting multi-scale features from the pseudo-image using an FPN network to generate a target feature map containing contextual information includes: The FPN network includes a first connection unit consisting of multiple cascaded ResUnit modules arranged from bottom to top, a second connection unit consisting of multiple cascaded CSPUnit modules arranged from top to bottom, and a third connection unit consisting of DenseNet connecting the first connection unit and the second connection unit. The pseudo-image is downsampled and multi-scale feature extraction is performed by the first connection unit to generate a base feature map. The base feature map is upsampled and multi-scale feature extraction is performed by the second connection unit to generate a reference feature map. The base feature map and the reference feature map are fused by the third connection unit to generate a target feature map containing context information. The ResUnit module uses the Swish function as the output activation function.

8. The 3D target detection method based on superpoints as described in claim 1, characterized in that, The step of decoupling the target feature map using a decoupling detection head to generate a decoupling result including confidence levels includes: The L×L window is slid across M superpoints according to the first step length. Each time the window is slid, the superpoints within the window are merged to generate m window points. Calculate the center point of each window point based on all the superpoints within each window point, and obtain m window centers; Calculate the Euclidean distance between each target point on the target feature map and all window points, and map the target point to the window point with the smallest Euclidean distance to obtain the target ground truth information; The target feature map is decoupled by decoupling the detection head and the target ground truth information to generate a decoupling result including confidence.

9. The 3D target detection method based on superpoints as described in claim 1, characterized in that, The decoupling results include target category and target location information.

10. A 3D target detection system based on superpoints, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 9.

Images