A 3D object detection method based on multi-scale difference allocation strategy

Abstract

The present invention proposes a three-dimensional target detection method based on a multi-scale difference allocation strategy, which relates to the field of 3D point cloud processing; specifically, the following steps are included: S1, introducing sparse tensors and point cloud voxelization operations for preprocessing: S2, sending the data preprocessed in step S1 to a sparse neural network 3D-MSDA, the sparse neural network includes the backbone network, neck and head of CNN; S3, designing a multi-scale difference allocation evaluator, and designing an evaluator strategy including coarse allocation and fine allocation; S4, calculating the loss function, and using the network of the loss function to perform target detection on the corresponding data set. In the present invention, the evaluator uses a set of global parameters to allocate points in the point cloud to different targets; for each scale, multiple evaluators are called on the basis of coarse allocation, and more detailed allocation is performed based on the feature map of a specific scale and the fine position information of the target, which helps to improve the performance of three-dimensional target detection when processing targets with multiple scales or multiple levels.

Description

Technical Field

[0001] The present invention relates to the field of 3D point cloud processing, and in particular to a three-dimensional target detection method based on a multi-scale difference allocation strategy. Background Art

[0002] With the advancement of science and technology and the development of society, the technologies of autonomous driving, robots, and augmented reality equipment are gradually moving from laboratories to public life. Autonomous driving vehicles rely on sensors to perceive the external environment to complete driving tasks, different robots have many functions such as intelligent sweeping and industrial cleaning, and augmented reality equipment shines in the field of games. Behind this, it is inseparable from the target detection tasks that escort these tasks.

[0003] Point cloud-based 3D object detection is a core technology that aims to accurately locate and identify targets based on a given 3D point set. This technology not only plays an important role in the field of autonomous driving, ensuring the vehicle's accurate perception of the surrounding environment and obstacle recognition, but also has broad and far-reaching applications in robotics, augmented reality (AR) and other fields. Common 2D data processing techniques can process densely arranged input images, but cannot be directly used for 3D object detection because 3D data is usually scanned or reconstructed into a set of points or grids. The sparse data obtained in this way only occupies the surface part or even a small part of the original object, which brings challenges to the processing technology of 3D object detection.

[0004] Previous 3D object detectors typically process sparse 3D data through two main methods. One method is to directly input a set of points into a multilayer perceptron; the other is to convert the sparse 3D data into a dense representation for 3D object detection. The multilayer perceptron method processes the scene in the form of a point set, but limits the number of points processed by the network, which may result in low-resolution input, which hinders expansion in large or high-resolution scenes and requires cropping and splicing the results, or using sliding windows for cropping and splicing, which limits the network's perception of larger contexts. The dense representation method uses volumetric features, but its memory and computational complexity are cubic, limiting the application of deep networks.

[0005] Therefore, a three-dimensional target detection method based on multi-scale difference allocation strategy is provided to solve the above problems. Summary of the invention

[0006] The present invention provides a three-dimensional target detection method based on a multi-scale difference allocation strategy, designs a hierarchical sparse tensor network 3D-MSDA, and achieves good detection results on the challenging indoor SUN RGB-D, ScanNet and S3DIS datasets. The multi-scale difference allocation strategy is introduced, a multi-scale difference allocation evaluator is designed, and the improvement of the perception ability of targets of different scales is demonstrated in experiments.

[0007] To achieve the above object, the present invention provides a three-dimensional target detection method based on a multi-scale difference allocation strategy, which specifically includes the following steps:

[0008] S1. Introduce sparse tensor and point cloud voxelization operations for preprocessing:

[0009] S2, sending the data preprocessed in step S1 to the sparse neural network 3D-MSDA, the sparse neural network includes the backbone network, neck and head of CNN;

[0010] S3. Design a multi-scale difference allocation evaluator, and design an evaluator strategy including coarse allocation and fine allocation;

[0011] S4. Calculate the loss function and use the network with the loss function to perform target detection on the corresponding dataset.

[0012] Preferably, in step S1, the following steps are specifically included:

[0013] S11: Sparse tensor is a data structure used to represent a high-dimensional array with zero elements. It is suitable for point cloud representation. A sparse tensor of order D is represented as is a D-dimensional array, represented by a sparse matrix. A sparse matrix is ​​represented as a set of non-zero coordinates. Where supp is a set-theoretic support operator, and the corresponding characteristic formula of the sparse matrix is for:

[0014]

[0015] in represents the D-th axis coordinate of the i-th non-zero element, f i Represents the features related to the i-th non-zero element. The information contained in the non-zero element is equivalent to a sparse tensor

[0016] S12: Point cloud voxelization is the process of converting a continuous three-dimensional space into a discrete voxel grid; the above process includes the following steps:

[0017] A: Select the size of the voxel and determine the resolution of the voxel grid;

[0018] B: Determine the boundaries of the voxel grid based on the spatial range of the point cloud data;

[0019] C: Assign point cloud data to voxels: traverse the points in the point cloud and calculate the voxel to which they belong based on the coordinates of the points.

[0020] The above operations can reduce the complexity of the data, making it easier to process and understand. In addition, voxelization can introduce a certain degree of spatial consistency in three-dimensional space, which also helps to detect objects.

[0021] Preferably, in step S2, the following steps are specifically included:

[0022] S21: A residual network ResNet modified by three-dimensional sparse convolution is used as the backbone network, named MEResNet, and pyramid context features are extracted through the downsampling module;

[0023] S22: The neck is the simplified GSDN decoder without the convolutional scoring layer in GSDN;

[0024] S23: The head of 3D-MSDA consists of two sparse convolutional layers. Except for the valuer, other weights are shared across layers. Multiple layers output classification probabilities and the bounding box regression parameter δ.

[0025] Preferably, in step S3, for the modification of the evaluator in the header, the center distance is used based on the evaluator. Limit the top k positions of each bounding box and redefine two parameters: one is the top point count threshold T on the bounding box p , define its value as τ, by increasing or decreasing the threshold T p , controls the number of points assigned to the target box; the second is the distance weight W used to adjust the point allocation result from the center of the bounding box d , define its value as ω, according to the distance weight W d The closer the point is to the center of the target frame, the higher the distance weight is, and the greater the probability of being assigned to the corresponding target frame. The inverse distance weight is selected, and its formula is:

[0026]

[0027] Among them, distance_weights ij Indicates the distance weight from point i to bounding box j, center_distances ij is the center distance between point i and bounding box j, distance_weight is the distance weight parameter, ∈=1e -6 ;

[0028] The process of assigning an evaluator includes:

[0029] Multiple levels of feature maps are obtained through different layers of Backbone. The feature maps have different spatial resolutions. A shared global assigner is created for each level of feature map for coarse allocation.

[0030] The coarse assignment of the assigner associates the points in the point cloud with the target box and assigns object IDs to the points in the point cloud. The assignment process is based on W d Parameter,allocation logic is: accept point cloud, target box and target label information, initialize the point level, number of points, number of target boxes and then splice the point cloud and target box, specifically:

[0031] Through the position level level_condition, determine the level of the point and match the target label;

[0032] By calculating the distance between the point and the center of the target box, determine whether the point is close to the center of the target box, and ensure that at most τ points are assigned to the target box;

[0033] Determine the point to be assigned to the corresponding target box. First, find the target box closest to the point, and then find the final assigned target box of the point based on the distance condition. Finally, return the assigned target box ID to determine the object to which the point belongs.

[0034] The fine allocation of the assigner adapts to the requirements of different scales according to different parameters. The MutillevelAssigner3D class uses multiple Assigner3D instances internally to perform the allocation.

[0035] Preferably, in step S4, the multi-level location allocation strategy allows the network to perform more accurate location allocation according to information of different levels or scales of the target;

[0036] Through the assignment of the evaluator in step S3, the position Compared with the ground truth bounding box Matching, location Therefore, the ground truth label and 3D center distance value Related, in the reasoning process, the score Multiply by the 3D center distance before NMS The final overall loss function is expressed as follows:

[0037]

[0038] Where, the matching position N pos The number is Classification loss L cls is the focal loss; the regression loss L reg The DIoU loss takes into account the distance between the center point of the bounding box and the size difference of the bounding box. The estimator considers the points near the bounding box. The center distance loss L cntr is the binary cross entropy loss, and for each loss, the predicted values ​​are represented by hats.

[0039] Therefore, the present invention adopts the above-mentioned three-dimensional target detection method based on the multi-scale difference allocation strategy, designs a hierarchical sparse tensor network 3D-MSDA, and achieves good detection results on the challenging indoor SUN RGB-D, ScanNet and S3DIS datasets. It introduces a multi-scale difference allocation strategy, designs a multi-scale difference allocation assigner, and proves the improvement of the perception ability of targets of different scales in experiments.

[0040] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flow chart of a three-dimensional target detection method based on a multi-scale difference allocation strategy of the present invention;

[0042] Figure 2 It is a network structure diagram of a three-dimensional target detection method based on a multi-scale difference allocation strategy of the present invention;

[0043] Figure 3 It is a network detection work visualization diagram of a three-dimensional target detection method based on a multi-scale difference allocation strategy of the present invention. DETAILED DESCRIPTION

[0044] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0045] Unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0046] Example

[0047] like Figure 1 As shown, the present invention provides a three-dimensional target detection method based on a multi-scale difference allocation strategy, which specifically includes the following steps:

[0048] S1. Introduce sparse tensor and point cloud voxelization operations for preprocessing; perform point cloud target detection on three datasets:

[0049] (1) The ScanNet dataset contains 1513 3D reconstructions of indoor scenes annotated with instance labels of 18 semantic classes. The training subset consists of 1201 scans, while the remaining 312 scans are used for validation.

[0050] (2) SUN RGB-D is a monocular 3D scene understanding dataset containing more than 10,000 indoor RGB-D images. Experiments are conducted on the 10 most common categories of objects. The training and validation splits contain 5285 and 5050 point clouds, respectively.

[0051] (3) The Stanford Large-Scale 3D Indoor Spaces (S3DIS) dataset contains 3D scans of 6 buildings with a total of 272 rooms, each with instances and semantic labels of 7 structural elements, such as floor and ceiling, and 5 furniture categories. Our method is trained and evaluated on the official furniture segmentation, and the most widely used Area 5 is used as the test segmentation for our method, of which 68 rooms are used for validation, while the remaining 204 rooms comprise the training subset.

[0052] In step S1, the following steps are specifically included:

[0053] S11: Sparse tensor is a data structure used to represent a high-dimensional array with zero elements. It is suitable for point cloud representation. A sparse tensor of order D is represented as is a D-dimensional array, represented by a sparse matrix. A sparse matrix is ​​represented as a set of non-zero coordinates. Where supp is a set-theoretic support operator, and the corresponding characteristic formula of the sparse matrix is for:

[0054]

[0055] in represents the D-th axis coordinate of the i-th non-zero element, f i Represents the features related to the i-th non-zero element. The information contained in the non-zero element is equivalent to a sparse tensor

[0056] S12: Point cloud voxelization is the process of converting a continuous three-dimensional space into a discrete voxel grid; the above process includes the following steps:

[0057] A: Select the size of the voxel and determine the resolution of the voxel grid;

[0058] B: Determine the boundaries of the voxel grid based on the spatial range of the point cloud data;

[0059] C: Assign point cloud data to voxels: traverse the points in the point cloud and calculate the voxel to which they belong based on the coordinates of the points.

[0060] The above operations can reduce the complexity of the data, making it easier to process and understand. In addition, voxelization can introduce a certain degree of spatial consistency in three-dimensional space, which also helps to detect objects.

[0061] S2, the data pre-processed in step S1 is sent to the sparse neural network 3D-MSDA. Figure 2 As shown, the sparse neural network includes the backbone network, neck and head of CNN; in step S2, the following steps are specifically included:

[0062] S21: A residual network ResNet modified by three-dimensional sparse convolution is used as the backbone network, named MEResNet, and pyramid context features are extracted through the downsampling module;

[0063] S22: The neck is the simplified GSDN decoder without the convolutional scoring layer in GSDN;

[0064] S23: The head of 3D-MSDA consists of two sparse convolutional layers. Except for the valuer, other weights are shared across layers. Multiple layers output classification probabilities and the bounding box regression parameter δ.

[0065] S3, design a multi-scale difference allocation evaluator, design an evaluator strategy including coarse allocation and fine allocation; in step S3, for the modification of the evaluator in the head, use the center distance on the basis of the evaluator Limit the top k positions of each bounding box and redefine two parameters: one is the top point count threshold T on the bounding box p , define its value as τ, by increasing or decreasing the threshold T p , controls the number of points assigned to the target box; the second is the distance weight W used to adjust the point allocation result from the center of the bounding box d , define its value as ω, according to the distance weight W d The closer the point is to the center of the target frame, the higher the distance weight is, and the greater the probability of being assigned to the corresponding target frame. The inverse distance weight is selected, and its formula is:

[0066]

[0067] Among them, distance_weights ij Indicates the distance weight from point i to bounding box j, center_distances ijis the center distance between point i and bounding box j, distance_weight is the distance weight parameter, ∈=1e -6 ;

[0068] The assignment process of the evaluator is as follows Figure 3 As shown, including:

[0069] Multiple levels of feature maps are obtained through different layers of Backbone. The feature maps have different spatial resolutions. A shared global assigner is created for each level of feature map for coarse allocation.

[0070] The coarse assignment of the assigner associates the points in the point cloud with the target box and assigns object IDs to the points in the point cloud. The assignment process is based on W d Parameter,allocation logic is: accept point cloud, target box and target label information, initialize the point level, number of points, number of target boxes and then splice the point cloud and target box, specifically:

[0071] Through the position level level_condition, determine the level of the point and match the target label;

[0072] By calculating the distance between the point and the center of the target box, determine whether the point is close to the center of the target box, and ensure that at most τ points are assigned to the target box;

[0073] Determine the point to be assigned to the corresponding target box. First, find the target box closest to the point, and then find the final assigned target box of the point based on the distance condition. Finally, return the assigned target box ID to determine the object to which the point belongs.

[0074] The fine allocation of the assigner adapts to the requirements of different scales according to different parameters. The MutillevelAssigner3D class uses multiple instances of Assigner3D to perform the allocation internally, and each instance allocates targets at different levels. It is an extension of Assigner3D, and its purpose is to support multi-scale allocation strategies to adapt to targets of different scales and sizes. Call multiple instances of Assigner3D to perform the allocation, and each level has its specific maximum top point threshold and distance weight parameters. The following are some descriptions of the refinement assigner:

[0075] (1) The MutillevelAssigner3D class is initialized by passing in the top_pts_thresholds, label2level, and distance_weights parameters, which represent the maximum top point threshold for each level, the mapping from labels to levels, and the distance weight for each level, respectively.

[0076] (2) In the initialization method, a list of multiple Assigner3D instances is created, each instance corresponding to a level, and the relevant parameters of the level are used.

[0077] (3) The assign method receives point cloud data, GroundTruth bounding box, GroundTruth label, and image meta information as input, and executes a multi-level assignment strategy. First, the point level information is extracted and the point cloud data is merged into a tensor. Then, the presence or absence of the GroundTruth label determines whether the assignment is required. Next, it loops through each level of the assigner and passes the features and the corresponding maximum top point count threshold to the assign method of each assigner. Finally, it stores the assignment results of each level in a list and returns it.

[0078] In summary, the core idea of ​​this part of the design is to apply different Assigner3Ds at different levels to perform assignments, so as to make full use of multi-scale and multi-level information to improve the performance of object detection. MutillevelAssigner3D uses multiple Assigner3D instances, and the influence between them is reflected in the consistency, synergy and parameter sharing of assignments. By adjusting the parameters of each level, the model of this scheme can flexibly control the assignment strategy of each level to meet the needs of practical applications.

[0079] S4, calculate the loss function, and use the network of the loss function to perform target detection on the corresponding data set. In step S4, the multi-level location allocation strategy allows the network to perform more accurate location allocation based on information of different levels or scales of the target;

[0080] Through the assignment of the evaluator in step S3, the position Compared with the ground truth bounding box Matching, location Therefore, the ground truth label and 3D center distance value Related, in the reasoning process, the score Multiply by the 3D center distance before NMS The final overall loss function is expressed as follows:

[0081]

[0082] Where, the matching position N pos The number is Classification loss L cls is the focal loss; the regression loss L regThe DIoU loss takes into account the distance between the center point of the bounding box and the size difference of the bounding box. The estimator considers the points near the bounding box. The center distance loss L cntr is the binary cross entropy loss, and for each loss, the predicted values ​​are represented by hats.

[0083] Test Example:

[0084] In terms of evaluation indicators, this scheme uses the mean average precision (mAP) on the test set, following the widely used convention for 2D object detection. When the 3D intersection-superunion (IoU) between the predicted and true bounding boxes exceeds a certain threshold, we regard the detection as a positive match. For this purpose, we choose two thresholds of IoU of 0.25 and 0.5, that is, the evaluation indicators are mAP@0.25 and mAP@0.5. This indicator can be expressed as:

[0085]

[0086] Where N is the total number of categories. i is the average precision of the i-th category. Table 1 shows the comparison results between this scheme and other schemes;

[0087] Table 1

[0088]

[0089]

[0090]

[0091] As can be seen from Table 1, compared with other methods, this method achieves comparable or even higher scores on the three point cloud object detection datasets, especially on the S3DIS dataset, which proves that this method is effective.

[0092] Therefore, the present invention adopts the above-mentioned three-dimensional target detection method based on the multi-scale difference allocation strategy, designs a hierarchical sparse tensor network 3D-MSDA, and achieves good detection results on the challenging indoor SUN RGB-D, ScanNet and S3DIS datasets. It introduces a multi-scale difference allocation strategy, designs a multi-scale difference allocation assigner, and proves the improvement of the perception ability of targets of different scales in experiments.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A three-dimensional object detection method based on a multi-scale difference allocation strategy, characterized in that: The specific steps include: S1. Introduce sparse tensor and point cloud voxelization operations for preprocessing: S2, sending the data preprocessed in step S1 to the sparse neural network 3D-MSDA, the sparse neural network includes the backbone network, neck and head of CNN; S3. Design a multi-scale difference allocation evaluator, and design an evaluator strategy including coarse allocation and fine allocation; In step S3, for the modification of the evaluator in the header, the center distance is used based on the evaluator Limit the top k positions of each bounding box and redefine two parameters: one is the top point count threshold T on the bounding box p , define its value as τ, by increasing or decreasing the threshold T p , controls the number of points assigned to the target box; the second is the distance weight W used to adjust the point allocation result from the center of the bounding box d , define its value as ω, according to the distance weight W d The closer the point is to the center of the target frame, the higher the distance weight is, and the greater the probability of being assigned to the corresponding target frame. The inverse distance weight is selected, and its formula is: Among them, distance_weights ij Indicates the distance weight from point i to bounding box j, center_distances ij is the center distance between point i and bounding box j, distance_weight is the distance weight parameter, ε = 1e -6 ; The process of assigning an evaluator includes: Multiple levels of feature maps are obtained through different layers of Backbone. The feature maps have different spatial resolutions. A shared global assigner is created for each level of feature map for coarse allocation. The coarse assignment of the assigner associates the points in the point cloud with the target box and assigns object IDs to the points in the point cloud. The assignment process is based on W d Parameter,allocation logic is: accept point cloud, target box and target label information, initialize the point level, number of points, number of target boxes and then splice the point cloud and target box, specifically: Through the position level level_condition, determine the level of the point and match the target label; By calculating the distance between the point and the center of the target box, determine whether the point is close to the center of the target box, and ensure that at most τ points are assigned to the target box; Determine the point to be assigned to the corresponding target box. First, find the target box closest to the point, and then find the final assigned target box of the point based on the distance condition. Finally, return the assigned target box ID to determine the object to which the point belongs. The fine-scale assignment of the assigner is adapted to the requirements of different scales according to different parameters. The MutillevelAssigner3D class uses multiple Assigner3D instances internally to perform the assignment; S4, calculate the loss function, and use the network of the loss function to perform target detection on the corresponding data set; In step S4, the multi-level location allocation strategy allows the network to perform more accurate location allocation based on information of different levels or scales of the target; Through the assignment of the evaluator in step S3, the position Compared with the ground truth bounding box Matching, location Therefore, the ground truth label and 3D center distance value Related, in the reasoning process, the score Multiply by the 3D center distance before NMS The final overall loss function is expressed as follows: Where, the matching position N pos The number is Classification loss L cls is the focal loss; the regression loss L reg The DIoU loss takes into account the distance between the center point of the bounding box and the size difference of the bounding box. The estimator considers the points near the bounding box. The center distance loss L cntr is the binary cross entropy loss, and for each loss, the predicted values ​​are represented by hats.

2. The three-dimensional target detection method based on the multi-scale difference allocation strategy according to claim 1 is characterized in that: In step S1, the following steps are specifically included: S11: Sparse tensor is a data structure used to represent a high-dimensional array with zero elements. It is suitable for point cloud representation. A sparse tensor of order D is represented as is a D-dimensional array, represented by a sparse matrix. A sparse matrix is ​​represented by a set of non-zero coordinates C=supp(T), where supp is a set-theoretic support operator. The corresponding characteristic formula of the sparse matrix T is: in represents the D-th axis coordinate of the i-th non-zero element, f i Represents the features related to the i-th non-zero element. The information contained in the non-zero element is equivalent to a sparse tensor S12: Point cloud voxelization is the process of converting a continuous three-dimensional space into a discrete voxel grid; the above process includes the following steps: A: Select the size of the voxel and determine the resolution of the voxel grid; B: Determine the boundaries of the voxel grid based on the spatial range of the point cloud data; C: Assign point cloud data to voxels: traverse the points in the point cloud and calculate the voxel to which they belong based on the coordinates of the points.

3. The three-dimensional target detection method based on a multi-scale difference allocation strategy according to claim 1, characterized in that: In step S2, the following steps are specifically included: S21: A residual network ResNet modified by three-dimensional sparse convolution is used as the backbone network, named MEResNet, and pyramid context features are extracted through the downsampling module; S22: The neck is the simplified GSDN decoder without the convolutional scoring layer in GSDN; S23: The head of 3D-MSDA consists of two sparse convolutional layers. Except for the valuer, other weights are shared across layers. Multiple layers output classification probabilities and the bounding box regression parameter δ.

Images