Method, device and computer-readable storage medium for detecting objects in point cloud data

Abstract

The present disclosure relates to a method, device, and computer-readable storage medium for detecting targets in point cloud data, and relates to the field of computer technology. The method of the present disclosure includes: inputting point cloud data into a point cloud feature extraction network to obtain multiple key points in the output point cloud data, as well as feature information of each key point; for each key point, encoding the feature information of the key point based on the correlation between the key point and other key points within a preset range of the key point to obtain a first feature code of the key point; classifying each key point, determining the point classified as the target center as the reference center point; for each reference center point, encoding the first feature code of the reference center point based on the correlation between the reference center point and other reference center points to obtain a second feature code of the reference center point; and predicting the position and category of each target in the point cloud data based on the second feature code of each reference center point.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method, device, and computer-readable storage medium for detecting an object in point cloud data. Background Art

[0002] 3D (3-dimensional) object detection aims to identify and locate objects in 3D point clouds. It has been widely used in fields such as autonomous driving and augmented reality. Compared to 2D images, 3D point clouds can provide the geometric shape of objects and capture the 3D structure of the scene. Summary of the Invention

[0003] The inventors discovered that due to the three-dimensional features and irregularity of point clouds, they cannot be directly processed using powerful deep learning models such as convolutional neural networks. Therefore, specialized 3D feature learning technology is required to identify targets in point cloud data.

[0004] A technical problem to be solved by the present disclosure is to propose a method for detecting targets in point cloud data to improve the accuracy of target detection in point cloud data.

[0005] According to some embodiments of the present disclosure, a method for detecting targets in point cloud data is provided, including: inputting point cloud data into a point cloud feature extraction network to obtain multiple key points in the output point cloud data, and feature information of each key point; for each key point, encoding the feature information of the key point according to the correlation between the key point and other key points within a preset range of the key point, to obtain a first feature code of the key point; classifying each key point, and determining the point classified as the target center as the reference center point; for each reference center point, encoding the first feature code of the reference center point according to the correlation between the reference center point and other reference center points, to obtain a second feature code of the reference center point; and predicting the position and category of each target in the point cloud data based on the second feature code of each reference center point.

[0006] In some embodiments, for each key point, the feature information of the key point is encoded according to the correlation between the key point and other key points within the preset range of the key point, and the first feature code of the key point is obtained, including: for each key point, according to the feature information of the key point, the feature information of other key points within the preset range of the key point, and the relative position relationship between the key point and other key points within the preset range of the key point, the first feature code of the key point is determined based on the self-attention mechanism.

[0007] In some embodiments, for each key point, according to the feature information of the key point, the feature information of other key points within the preset range of the key point, and the relative position relationship between the key point and other key points within the preset range of the key point, determining the first feature encoding of the key point based on the self-attention mechanism includes: for each key point, inputting the feature information and position information of the key point, and the feature information and position information of other key points within the preset range of the key point into the first self-attention module of the encoder in the first conversion model; in the first self-attention module, taking the other key points within the preset range of the key point as relative points, and for each relative point, respectively inputting the position information of the key point and the position information of the relative point. The first position encoding layer, the second position encoding layer and the third position encoding layer are input to determine the first relative position encoding, the second relative position encoding and the third relative position encoding of the key point and the relative point; the key vector and the value vector of the relative point are determined according to the product of the feature information of the relative point and the key matrix and the value matrix in the first self-attention module; the query vector of the key point is determined according to the product of the feature information of the key point and the query matrix in the first self-attention module; the first feature encoding of the key point is determined according to the first relative position encoding, the second relative position encoding and the third relative position encoding of the key point and each relative point, the key vector of each relative point, the value vector of each relative point and the query vector of the key point.

[0008] In some embodiments, determining the first feature code of the key point based on the first relative position code, the second relative position code and the third relative position code of the key point and each relative point, the key vector of each relative point, the value vector of each relative point and the query vector of the key point includes: for each relative point, taking the sum of the first position code of the key point and the relative point and the query vector of the key point as the revised query vector of the key point; taking the sum of the second position code of the key point and the relative point and the key vector of the relative point as the revised key vector of the relative point; taking the sum of the third position code of the key point and the relative point and the value vector of the relative point as the revised value vector of the relative point; inputting the product of the revised query vector of the key point and the revised key vector of the relative point and the dimension of the feature information of the key point into the first normalization layer to obtain the weight of the relative point; and performing weighted summation on the revised value vectors of each relative point according to the weight of each relative point to obtain the first feature code of the key point.

[0009] In some embodiments, the first position encoding layer, the second position encoding layer and the third position encoding layer are respectively the first feedforward network, the second feedforward network and the third feedforward network, and inputting the position information of the key point and the position information of the relative point into the first position encoding layer, the second position encoding layer and the third position encoding layer respectively includes: inputting the difference between the coordinates of the key point and the coordinates of the relative point into the first feedforward network, the second feedforward network and the third feedforward network respectively.

[0010] In some embodiments, for each key point, the feature information of the key point is encoded according to the correlation between the key point and other key points within the preset range of the key point, and the first feature code of the key point is obtained, including: for each key point, according to the feature information of the key point, the feature information of other key points within the preset range of the key point, the relative position relationship between the key point and other key points within the preset range of the key point, and the relative geometric structure relationship between the key point and other key points within the preset range of the key point, the first feature code of the key point is determined based on the self-attention mechanism.

[0011] In some embodiments, for each key point, according to the feature information of the key point, the feature information of other key points within the preset range of the key point, the relative position relationship between the key point and other key points within the preset range of the key point, and the relative geometric structure relationship between the key point and other key points within the preset range of the key point, determining the first feature encoding of the key point based on the self-attention mechanism includes: for each key point, inputting the feature information, position information and geometric structure information of the key point, and the feature information, position information and geometric structure information of other key points within the preset range of the key point into the first self-attention module of the encoder in the first conversion model; in the first self-attention module, taking the other key points within the preset range of the key point as relative points, and for each relative point, inputting the position information of the key point and the position information of the relative point into the first position encoding layer, The second position encoding layer and the third position encoding layer determine the first relative position encoding, the second relative position encoding and the third relative position encoding of the key point and the relative point; the geometric structure information of the key point and the geometric structure information of the relative point are input into the geometric structure encoding layer to determine the relative geometric structure weight of the key point and the relative point; the key vector and the value vector of the relative point are determined according to the product of the feature information of the relative point and the key matrix and the value matrix in the first self-attention module respectively; the query vector of the key point is determined according to the product of the feature information of the key point and the query matrix in the first self-attention module; the first feature encoding of the key point is determined according to the first relative position encoding, the second relative position encoding, the third relative position encoding and the relative geometric structure weight of the key point and each relative point, the key vector of each relative point, the value vector of each relative point and the query vector of the key point.

[0012] In some embodiments, determining the first feature code of the key point based on the first relative position code, the second relative position code, the third relative position code and the relative geometric structure weight of the key point and each relative point, the key vector of each relative point, the value vector of each relative point and the query vector of the key point includes: for each relative point, taking the sum of the first position code of the key point and the relative point and the query vector of the key point as the revised query vector of the key point; taking the sum of the second position code of the key point and the relative point and the key vector of the relative point as the revised key vector of the relative point; taking the sum of the third position code of the key point and the relative point and the value vector of the relative point as the revised value vector of the relative point; inputting the product of the revised query vector of the key point and the revised key vector of the relative point, the relative geometric structure weight of the key point and the relative point and the dimension of the feature information of the key point into the first normalization layer to obtain the weight of the relative point; and performing weighted summation on the revised value vectors of each relative point according to the weight of each relative point to obtain the first feature code of the key point.

[0013] In some embodiments, the product of the modified query vector of the key point and the modified key vector of the relative point, the relative geometric structure weight of the key point and the relative point, and the dimension of the feature information of the key point are input into the first normalization layer, and the weight of the relative point is obtained, which includes: dividing the product of the modified query vector of the key point and the modified key vector of the relative point by the square root of the dimension of the feature information of the key point, and then superimposing it with the relative geometric structure weight of the key point and the relative point, and inputting the result into the first normalization layer to obtain the weight of the relative point.

[0014] In some embodiments, the geometric structure information includes: at least one of the normal vector of the local plane where it is located and the radius of curvature of the local plane where it is located. Determining the relative geometric structure weight of the key point and the relative point includes: determining the relative geometric structure weight of the key point and the relative point based on at least one of the distance between the key point and the relative point, the dot product of the normal vector of the local plane where the key point is located and the normal vector of the local plane where the relative point is located, the difference between the radius of curvature of the local plane where the key point is located and the radius of curvature of the local plane where the relative point is located, and the angle between the normal vector of the local plane where the key point is located and the normal vector of the local plane where the relative point is located.

[0015] In some embodiments, the relative geometric structure weight of the key point and the relative point decreases as the distance between the key point and the relative point increases; the relative geometric structure weight of the key point and the relative point increases as the dot product of the normal vector of the local plane where the key point is located and the normal vector of the local plane where the relative point is located increases; the relative geometric structure weight of the key point and the relative point increases as the difference between the curvature radius of the local plane where the key point is located and the curvature radius of the local plane where the relative point is located increases; the relative geometric structure weight of the key point and the relative point increases as the angle between the normal vector of the local plane where the key point is located and the normal vector of the local plane where the relative point is located increases after passing through the feature propagation layer.

[0016] In some embodiments, for each reference center point, the first feature code of the reference center point is encoded according to the correlation between the reference center point and other reference center points, and the second feature code of the reference center point is obtained, including: for each reference center point, according to the first feature code of the reference center point, the first feature codes of other reference center points, and the relative position relationship between the reference center point and other reference center points, the second feature code of the reference center point is determined based on the self-attention mechanism.

[0017] In some embodiments, for each reference center point, according to the first feature code of the reference center point, the first feature code of other reference center points, and the relative position relationship between the reference center point and other reference center points, determining the second feature code of the reference center point based on the self-attention mechanism includes: for each reference center point, inputting the first feature code and position information of the reference center point, and the first feature code and position information of other reference center points into the second self-attention module of the encoder in the second conversion model; in the second self-attention module, taking other reference center points as relative center points, and for each relative center point, inputting the position information of the reference center point and the position information of the relative center point into the fourth position encoding layer, the fifth position encoding layer, and the fourth position encoding layer, respectively. The coding layer and the sixth position coding layer determine the fourth relative position coding, the fifth relative position coding and the sixth relative position coding of the reference center point and the relative center point; determine the key vector and the value vector of the relative center point according to the product of the feature information of the relative center point and the key matrix and the value matrix in the second self-attention module; determine the query vector of the reference center point according to the product of the feature information of the reference center point and the query matrix in the second self-attention module; determine the second feature coding of the reference center point according to the fourth relative position coding, the fifth relative position coding and the sixth relative position coding of the reference center point and each relative center point, the key vector of each relative center point, the value vector of each relative center point and the query vector of the reference center point.

[0018] In some embodiments, determining the second feature code of the reference center point based on the fourth relative position code, the fifth relative position code and the sixth relative position code of the reference center point and each relative center point, the key vector of each relative center point, the value vector of each relative center point and the query vector of the reference center point includes: for each relative center point, taking the sum of the fourth position code of the reference center point and the relative center point and the query vector of the reference center point as the revised query vector of the reference center point; taking the sum of the fifth position code of the reference center point and the relative center point and the key vector of the relative center point as the revised key vector of the relative center point; taking the sum of the sixth position code of the reference center point and the relative center point and the value vector of the relative center point as the revised value vector of the relative center point; inputting the product of the revised query vector of the reference center point and the revised key vector of the relative center point and the dimension of the first feature code of the reference center point into the second normalization layer to obtain the weight of the relative center point; and performing weighted summation on the revised value vectors of each relative center point according to the weight of each relative center point to obtain the second feature code of the reference center point.

[0019] In some embodiments, the fourth relative position encoding, the fifth relative position encoding and the sixth relative position encoding are respectively the fourth feedforward network, the fifth feedforward network and the sixth feedforward network, and inputting the position information of the reference center point and the position information of the relative center point into the fourth position encoding layer, the fifth position encoding layer and the sixth position encoding layer respectively includes: inputting the difference between the coordinates of the reference center point and the coordinates of the relative center point into the fourth feedforward network, the fifth feedforward network and the sixth feedforward network respectively.

[0020] In some embodiments, each key point is classified and the point classified as the target center is determined as the reference center point, including: for each key point, the first feature encoding of the key point is input into the classification network to obtain the classification result of the key point; and whether the key point is the target center point according to the classification result.

[0021] In some embodiments, the classification network is trained using the position information of each key point with labeled information as training data, wherein, for each key point, when the key point is within the bounding box of a target and is the point closest to the center of the target, the labeled information of the key point is the point of the center of the target.

[0022] In some embodiments, based on the second feature code of each reference center point, predicting the position and category of each target in the point cloud data includes: inputting the second feature code of each reference center point into the decoder in the second conversion model to obtain the feature vector of each reference center point; inputting the feature vector of each reference center point into the target detection network to obtain the position and category of each target in the point cloud data.

[0023] In some embodiments, for each key point, other key points within the preset range of the key point are determined using the following method: for each key point, other key points are sorted from small to large according to their distance from the key point, and a preset number of other key points are selected in order from front to back as other key points within the preset range of the key point.

[0024] According to some other embodiments of the present disclosure, a device for detecting targets in point cloud data is provided, including: a feature extraction module for inputting point cloud data into a point cloud feature extraction network to obtain multiple key points in the output point cloud data, and feature information of each key point; a first encoding module for encoding the feature information of each key point according to the correlation between the key point and other key points within a preset range of the key point, to obtain a first feature code of the key point; a classification module for classifying each key point, and determining a point classified as a target center as a reference center point; a second encoding module for encoding the first feature code of each reference center point according to the correlation between the reference center point and other reference center points, to obtain a second feature code of the reference center point; and a target detection module for predicting the position and category of each target in the point cloud data according to the second feature code of each reference center point.

[0025] According to some further embodiments of the present disclosure, a device for detecting targets in point cloud data is provided, comprising: a processor; and a memory coupled to the processor, for storing instructions, which, when executed by the processor, causes the processor to execute a method for detecting targets in point cloud data as described in any of the aforementioned embodiments.

[0026] According to some further embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, wherein when the program is executed by a processor, the steps of the method for detecting an object in point cloud data of any of the aforementioned embodiments are implemented.

[0027] According to some further embodiments of the present disclosure, an article sorting device is provided, comprising: a detection device for targets in point cloud data of any of the aforementioned embodiments and a sorting component; the sorting component is used to sort targets according to the position and category of each target in the point cloud data output by the detection device for targets in the point cloud data.

[0028] In some embodiments, the device further includes: a point cloud acquisition component for acquiring point cloud data of a preset area and sending the point cloud data to a detection device for a target in the point cloud data.

[0029] The present disclosure extracts key points from point cloud data and feature information of each key point. For each key point, a first feature code of the key point is determined based on the correlation between the key point and other key points within a preset range of the key point. The first feature code reflects the correlation between points in a local area within the target. The key points are further divided into target center points and non-target center points. The target center point is used as a reference center point. For each reference center point, a second feature code of the reference center point is determined based on the correlation between the reference center point and other reference center points. The second feature code adds the correlation between targets to the correlation between points in the local area within the target. Then, based on the second feature code of each reference center point, the position and category of each target in the point cloud data are predicted. The solution of the present disclosure no longer models the correlation between all points in the point cloud, but instead divides the correlation between points into intra-target correlation and inter-target correlation. It can simultaneously capture local and global dependencies in the point cloud, adapt to the three-dimensional characteristics and irregularities of the point cloud data, improve the accuracy of target detection in the point cloud data, and improve detection efficiency and save computing costs.

[0030] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0032] Figure 1 A schematic flow chart illustrating a method for detecting an object in point cloud data according to some embodiments of the present disclosure is provided.

[0033] Figure 2 Schematic diagrams showing target models in point cloud data according to other embodiments of the present disclosure.

[0034] Figure 3 A schematic structural diagram of a device for detecting targets in point cloud data according to some embodiments of the present disclosure is shown.

[0035] Figure 4 A schematic structural diagram of an apparatus for detecting an object in point cloud data according to some other embodiments of the present disclosure is shown.

[0036] Figure 5 A schematic structural diagram of a device for detecting targets in point cloud data according to some further embodiments of the present disclosure is shown.

[0037] Figure 6 A schematic structural diagram of an item sorting device according to some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0039] The present invention provides a method for detecting targets in point cloud data. Figure 1 Provide a description.

[0040] Figure 1 Flowcharts of some embodiments of the method for detecting targets in point cloud data disclosed herein. Figure 1 As shown, the method of this embodiment includes steps S102 to S110.

[0041] In step S102, the point cloud data is input into a point cloud feature extraction network to obtain a plurality of key points in the output point cloud data and feature information of each key point.

[0042] Given a point cloud of N points with XYZ coordinates as input, the point cloud feature extraction network can downsample the point cloud data and learn the deep features of each point, thereby outputting a subset of points, and each point is represented by a C (C is a positive integer) dimensional feature, and these points are regarded as key points.

[0043] Point cloud feature extraction networks such as VoxelNet, PointNet, PointNet++, and 3DSSD are not limited to the examples given, and are used to extract key points and feature information of key points in point cloud data. For example, the PointNet++ network is used as a point cloud feature extraction network. The point cloud data containing N points is taken as input, and following the encoder-decoder structure, the input point cloud is first downsampled to 8 times the resolution (i.e., N / 8 points) through 4 set abstraction layers, and then upsampled to 2 times the resolution (i.e., N / 2 points) through the feature propagation layer, and each point is represented by a C-dimensional feature. The set of key points is represented as f i The feature information of the i-th key point is represented as a feature vector. The number of key points and the sampling method are not limited to the above examples and are determined based on the actual application model and test results.

[0044] In step S104 , for each key point, feature information of the key point is encoded according to the correlation between the key point and other key points within a preset range of the key point to obtain a first feature code of the key point.

[0045] For example, for each key point, other key points are sorted from smallest to largest according to their distance from the key point, and a preset number of other key points are selected in descending order of the sorting order as other key points within the preset range of the key point. For example, for each key point, the K key points closest to it are selected as other key points within the corresponding preset range (local area).

[0046] In some embodiments, for each key point, the first feature code of the key point is determined based on the self-attention mechanism according to the feature information of the key point, the feature information of other key points within the preset range of the key point, and the relative positional relationship between the key point and other key points within the preset range of the key point. Based on the self-attention mechanism, the importance or contribution of other key points within the preset range of the key point relative to the key point code can be determined. When encoding the key point, the features of the key point are described in combination with the features of other key points within the preset range, which can improve the accuracy of the feature expression of the key point. In addition, the relative positional relationship between the key point and other key points within the preset range of the key point is introduced into the attention mechanism, which improves the accuracy of the feature expression and thereby improves the accuracy of target detection.

[0047] Further, in some embodiments, for each key point, the feature information and position information of the key point, and the feature information and position information of other key points within a preset range of the key point are input into the first self-attention module of the encoder in the first conversion model; in the first self-attention module, the other key points within the preset range of the key point are used as relative points, and for each relative point, the position information of the key point and the position information of the relative point are respectively input into the first position encoding layer, the second position encoding layer and the third position encoding layer to determine the first relative position encoding, the second relative position encoding and the third relative position encoding of the key point and the relative point; according to the product of the feature information of the relative point and the key matrix and the value matrix in the first self-attention module, the key vector and the value vector of the relative point are determined; according to the product of the feature information of the key point and the query matrix in the first self-attention module, the query vector of the key point is determined; according to the first relative position encoding, the second relative position encoding and the third relative position encoding of the key point and each relative point, the key vector of each relative point, the value vector of each relative point and the query vector of the key point, the first feature encoding of the key point is determined.

[0048] Further, in some embodiments, for each relative point, the sum of the first position code of the key point and the relative point and the query vector of the key point is used as the revised query vector of the key point; the sum of the second position code of the key point and the relative point and the key vector of the relative point is used as the revised key vector of the relative point; the sum of the third position code of the key point and the relative point and the value vector of the relative point is used as the revised value vector of the relative point; the product of the revised query vector of the key point and the revised key vector of the relative point and the dimension of the feature information of the key point are input into the first normalization layer to obtain the weight of the relative point; the revised value vectors of each relative point are weightedly summed according to the weight of each relative point to obtain the first feature code of the key point.

[0049] For example, the first conversion model is a Transformer model. Since the first conversion model is used to determine the correlation between the internal points of the target, it can be called a Local Transformer. The first conversion model may include the encoder part in the Transformer. In addition, for example, the first position encoding layer, the second position encoding layer, and the third position encoding layer are respectively the first feedforward network (FFN), the second feedforward network, and the third feedforward network. The difference between the coordinates of the key point and the coordinates of the relative point is input into the first feedforward network, the second feedforward network, and the third feedforward network respectively to determine the first relative position encoding, the second relative position encoding, and the third relative position encoding of the key point and the relative point. The first normalization layer is, for example, a softmax layer.

[0050] For example, and Represents the feature vector and position coordinates (x i is a vector). In Local Transformer, for any key point x i , select the K key points closest to it as its corresponding local area, and input these points into a LocalTransformer module to model all key points belonging to the same local area:

[0051]

[0052] In formula (1) The output of the first self-attention module of the Local Transformer encoder. The encoder may also include FFN (feedforward neural network) after the first self-attention module. If FFN is not included after the first self-attention module, then can represent the first feature code, otherwise, The output after FFN represents the first feature code; is the query matrix in the first self-attention module, is the key matrix in the first self-attention module, is the median matrix of the first self-attention module, C is the dimension of the feature information of the key point, Represent the functions corresponding to the first position coding layer, the second position coding layer and the third position coding layer respectively. i ,x j ) represents x i ,x j The relative position encoding can be expressed by the following formula:

[0053] PE(x i ,x j )=FFN(x i -x j )=(x i -x j )W PE +b PE (2)

[0054] W PE and b PE represents the parameters of FFN (feedforward network or feature propagation layer), Corresponding W PE and b PE different.

[0055] A multi-head attention mechanism can be applied in the encoder. Each attention head can refer to the above formulas (1) and (2) to determine the encoding of the key point. The encoding of each attention head is concatenated and multiplied with the preset matrix (or the product is further passed through FFN) to obtain the first feature encoding of the key point. The query matrix, key matrix, value matrix, and parameters of the first position encoding layer, second position encoding layer, and third position encoding layer in each attention head are different.

[0056] The first feature encoding output by Local Transformer contains the contextual information of the local area where the key point is located, that is, the correlation between the internal points and points of the target.

[0057] In step S106, each key point is classified, and the point classified as the target center is determined as the reference center point.

[0058] In some embodiments, for each key point, the first feature code of the key point is input into the classification network to obtain a classification result of the key point; and whether the key point is the point of the target center is determined based on the classification result.

[0059] In some embodiments, the classification network is trained using the position information of each key point with labeled information as training data, wherein, for each key point, when the key point is within the bounding box of a target and is the point closest to the center of the target, the labeled information of the key point is the point of the center of the target.

[0060] The key points output by the first conversion model are dense points. Not every point represents a separate target (object). In order to reduce the redundancy of the final detection results, all key points are screened and only the key points located at the center of the target are retained. Therefore, it is necessary to determine whether each key point is the true target center. During the training process, a label is assigned to each key point. If a key point is within the bounding box of a target and is the closest point to the target center, it is assigned a positive label, otherwise it is assigned a negative label. A two-category network is trained based on the labels of the key points. During the testing process, all key points are input into the two-category network, and only the key points with positive classification results are retained as reference center points.

[0061] In step S108 , for each reference center point, the first feature code of the reference center point is encoded according to the correlation between the reference center point and other reference center points to obtain a second feature code of the reference center point.

[0062] In some embodiments, for each reference center point, the second feature code of the reference center point is determined based on the self-attention mechanism according to the first feature code of the reference center point, the first feature codes of other reference center points, and the relative position relationship between the reference center point and other reference center points.

[0063] Further, in some embodiments, for each reference center point, the first feature code and position information of the reference center point, and the first feature code and position information of other reference center points are input into the second self-attention module of the encoder in the second conversion model; in the second self-attention module, the other reference center points are used as relative center points, and for each relative center point, the position information of the reference center point and the position information of the relative center point are input into the fourth position coding layer, the fifth position coding layer and the sixth position coding layer respectively to determine the fourth relative position code, the fifth relative position code and the sixth relative position code of the reference center point and the relative center point; according to the product of the feature information of the relative center point and the key matrix and the value matrix in the second self-attention module, the key vector and the value vector of the relative center point are determined; according to the product of the feature information of the reference center point and the query matrix in the second self-attention module, the query vector of the reference center point is determined; according to the fourth relative position code, the fifth relative position code and the sixth relative position code of the reference center point and each relative center point, the key vector of each relative center point, the value vector of each relative center point and the query vector of the reference center point, the second feature code of the reference center point is determined.

[0064] Further, in some embodiments, for each relative center point, the sum of the fourth position code of the reference center point and the relative center point and the query vector of the reference center point is used as the revised query vector of the reference center point; the sum of the fifth position code of the reference center point and the relative center point and the key vector of the relative center point is used as the revised key vector of the relative center point; the sum of the sixth position code of the reference center point and the relative center point and the value vector of the relative center point is used as the revised value vector of the relative center point; the product of the revised query vector of the reference center point and the revised key vector of the relative center point and the dimension of the first feature code of the reference center point are input into the second normalization layer to obtain the weight of the relative center point; the revised value vectors of each relative center point are weightedly summed according to the weight of each relative center point to obtain the second feature code of the reference center point.

[0065] For example, the second conversion model is a Transformer model. Since the second conversion model is used to determine the correlation between targets, it can be called a Global Transformer. The second conversion model may include the encoder and decoder parts in the Transformer. In addition, the fourth relative position encoding, the fifth relative position encoding, and the sixth relative position encoding are respectively the fourth feedforward network, the fifth feedforward network, and the sixth feedforward network. The difference between the coordinates of the reference center point and the coordinates of the relative center point is input into the fourth feedforward network, the fifth feedforward network, and the sixth feedforward network, respectively, to determine the fourth relative position encoding, the fifth relative position encoding, and the sixth relative position encoding of the reference center point and the relative center point. The second normalization layer is, for example, a softmax layer.

[0066] For example, M reference center points are obtained from the key points, and Global Transformer aims to learn the correlation between these M different targets. Specifically, the feature set of M reference center points (for example, M reference center points) is represented as ) is input into the Global Transformer module to model the correlation between different targets:

[0067]

[0068] In formula (3), h i is the output of the second self-attention module of the Global Transformer encoder. The encoder may also include FFN after the second self-attention module. If FFN is not included after the second self-attention module, then h i can represent the second feature code, otherwise, h i The output after FFN represents the second feature code; is the query matrix in the second self-attention module, is the key matrix in the second self-attention module, is the median matrix of the second self-attention module, C is the dimension of the feature information of the key points, They respectively represent the fourth position coding layer, the fourth position coding layer and the function corresponding to the fourth position coding layer. The specific form of can refer to formula (2).

[0069] The multi-head attention mechanism can also be applied to the encoder of Global Transformer, which will not be described in detail here. i is the high-level feature of the output i-th reference center, which includes both the correlation of the internal points of the target and the correlation between different targets.

[0070] In step S110 , the position and category of each target in the point cloud data are predicted based on the second feature code of each reference center point.

[0071] In some embodiments, the second feature code of each reference center point is input into the decoder in the second conversion model to obtain the feature vector of each reference center point; the feature vector of each reference center point is input into the target detection network to obtain the position and category of each target in the point cloud data.

[0072] The object detection network is, for example, FFN, which determines the position and category of each object in the point cloud data based on a feature vector that includes the correlation between points inside the object and the correlation between different objects.

[0073] In the above embodiment, key points of point cloud data and feature information of each key point are extracted. For each key point, a first feature code is determined based on the correlation between the key point and other key points within a preset range of the key point. The first feature code reflects the correlation between points in a local area within the target. The key points are further divided into target center points and non-target center points, with the target center point serving as a reference center point. For each reference center point, a second feature code is determined based on the correlation between the reference center point and other reference center points. The second feature code adds the correlation between targets to the correlation between points in the local area within the target. Based on the second feature code of each reference center point, the position and category of each target in the point cloud data are predicted. The solution of the above embodiment no longer models the correlation between all points in the point cloud, but instead divides the correlation between points into intra-target correlation and inter-target correlation. This can simultaneously capture local and global dependencies in the point cloud, adapt to the three-dimensional characteristics and irregularities of the point cloud data, improve the accuracy of target detection in the point cloud data, and improve detection efficiency and save computational costs.

[0074] To further improve the accuracy of object detection in point cloud data, the above-mentioned embodiments have been improved. The inventors further explored the three-dimensional features of point cloud data and incorporated the geometric structure characteristics between points into the encoding process, making the learning of point cloud data features more accurate, thereby improving the accuracy of object detection. A specific embodiment is described below.

[0075] Regarding step S104, in some embodiments, for each key point, the first feature code of the key point is determined based on the self-attention mechanism according to the feature information of the key point, the feature information of other key points within the preset range of the key point, the relative positional relationship between the key point and other key points within the preset range of the key point, and the relative geometric structure relationship between the key point and other key points within the preset range of the key point.

[0076] Furthermore, in some embodiments, for each key point, the feature information, position information and geometric structure information of the key point, and the feature information, position information and geometric structure information of other key points within the preset range of the key point are input into the first self-attention module of the encoder in the first conversion model; in the first self-attention module, the other key points within the preset range of the key point are used as relative points, and for each relative point, the position information of the key point and the position information of the relative point are input into the first position encoding layer, the second position encoding layer and the third position encoding layer respectively to determine the first relative position code, the second relative position code and the third relative position code of the key point and the relative point; The geometric structure information and the geometric structure information of the relative point are input into the geometric structure encoding layer to determine the relative geometric structure weight of the key point and the relative point; the key vector and value vector of the relative point are determined according to the product of the feature information of the relative point and the key matrix and the value matrix in the first self-attention module respectively; the query vector of the key point is determined according to the product of the feature information of the key point and the query matrix in the first self-attention module; the first feature code of the key point is determined according to the first relative position code, second relative position code, third relative position code and relative geometric structure weight of the key point and each relative point, the key vector of each relative point, the value vector of each relative point and the query vector of the key point.

[0077] Further, in some embodiments, for each relative point, the sum of the first position code of the key point and the relative point and the query vector of the key point is used as the revised query vector of the key point; the sum of the second position code of the key point and the relative point and the key vector of the relative point is used as the revised key vector of the relative point; the sum of the third position code of the key point and the relative point and the value vector of the relative point is used as the revised value vector of the relative point; the product of the revised query vector of the key point and the revised key vector of the relative point, the relative geometric structure weight of the key point and the relative point, and the dimension of the feature information of the key point are input into the first normalization layer to obtain the weight of the relative point; the revised value vectors of each relative point are weightedly summed according to the weight of each relative point to obtain the first feature code of the key point.

[0078] Furthermore, in some embodiments, the product of the modified query vector of the key point and the modified key vector of the relative point is divided by the square root of the dimension of the feature information of the key point, and then superimposed with the relative geometric structure weight of the key point and the relative point. The result is input into the first normalization layer to obtain the weight of the relative point.

[0079] For example, the geometric structure information includes at least one of the normal vector of the local plane in which the key point is located and the radius of curvature of the local plane in which the key point is located. In some embodiments, the relative geometric structure weight of the key point and the relative point is determined based on at least one of the distance between the key point and the relative point, the dot product of the normal vector of the local plane in which the key point is located and the normal vector of the local plane in which the relative point is located, the difference between the radius of curvature of the local plane in which the key point is located and the radius of curvature of the local plane in which the relative point is located, and the angle between the normal vector of the local plane in which the key point is located and the normal vector of the local plane in which the relative point is located.

[0080] Further, in some embodiments, the relative geometric structure weight of the key point and the relative point decreases as the distance between the key point and the relative point increases; the relative geometric structure weight of the key point and the relative point increases as the dot product of the normal vector of the local plane where the key point is located and the normal vector of the local plane where the relative point is located increases; the relative geometric structure weight of the key point and the relative point increases as the difference between the curvature radius of the local plane where the key point is located and the curvature radius of the local plane where the relative point is located increases; the relative geometric structure weight of the key point and the relative point increases as the angle between the normal vector of the local plane where the key point is located and the normal vector of the local plane where the relative point is located increases after passing through the feature propagation layer.

[0081] For example, the first conversion model is a Transformer model, which is referred to as a Local Transformer. The first position encoding layer, the second position encoding layer, and the third position encoding layer can be a first feedforward network (FFN), a second feedforward network, and a third feedforward network, respectively, and the first normalization layer can be a softmax layer.

[0082] The relative geometric structure relationship between the key point and other key points within the preset range of the key point can be added to the attention mechanism. Specifically, formula (1) can be improved as follows:

[0083] In formula (4), G i,j It represents the relative geometric weight of key points i and j, which can be determined by the following formula:

[0084]

[0085] In formula (5), n i , n j Represents the normal vector of the local plane where the key points i and j are located, c i , c j Respectively represent the curvature radius of the local plane where the key points i and j are located, G represents the angle between the normal vector of the local plane where key point i is located and the normal vector of the local plane where key point j is located. β1, β2 and β3 are the parameters of the geometric structure encoding layer. FFN is a feedforward neural network or feature propagation layer. i,j It is a Gaussian function model that calculates the strength of the correlation between two points through geometric parameters such as the local plane normal vector, the local curvature radius, and the normal vector angle. The stronger the correlation between the two points, the stronger the corresponding Gaussian weight G i,j The bigger it is.

[0086] In some embodiments, for a key point, find the N points in the neighborhood that are closest to it, and then use the least squares method to find a plane so that the sum of the distances of these N points projected onto this plane is minimized. This plane is the local plane.

[0087] The method of the above embodiment adds an original relative geometric structure weight to express the geometric structure relationship between points, integrates object geometric features such as local plane normal vector, local curvature radius, and normal vector angle into the self-attention mechanism, and designs an efficient feature extraction model and target detection model specifically for processing point cloud data.

[0088] The following combination Figure 2 Some application examples of the present disclosure are described.

[0089] like Figure 2 As shown, the point cloud data is input into the point cloud feature extraction network (Point Cloud Backbone) to obtain the feature information of the key points (Point Feature), and then the feature information of the key points is input into the Local-Global Transformer model. In the Local-Global Transformer model, the feature information and position information of each key point and the key points in the local area of ​​the key point are input into the Local Transformer module, and the geometric structure information of the key point is input into the Local Transformer module to obtain the first feature code. The reference center point of each key point is selected through a classification network (for example, including a Sampling / Pooling module), and the first feature code and position information of the reference center point are input into the Global Transformer module to obtain the second feature code. The second feature code is input into the FFN to obtain the bounding box and category of the target.

[0090] The above embodiment proposes an end-to-end 3D point cloud object detection network based on the Transformer model, which can be called 3DTrans. It takes a 3D point cloud as input and outputs a set of labeled 3D bounding boxes to represent the location of the target (object). The overall structure of the 3DTrans detection network is as follows: Figure 2 As shown in the figure, it consists of two main components: a feature extraction network and a Local-Global Transformer. Given a point cloud of N points with X, Y, and Z coordinates as input, the feature extraction network downsamples the point cloud and learns deep features for each point, outputting a subset of points, each represented by a C-dimensional feature. These points are considered keypoints. The Local-Global Transformer takes the features of these keypoints as input and outputs the final object detection results. This improves the traditional Transformer model in two ways, making it more suitable for processing 3D point cloud data. First, rather than directly modeling the associations between all keypoints, the associations between points are divided into intra-object associations and inter-object associations. Specifically, the Local Transformer module learns the associations between points in a local region within the same object, while the Global Transformer module learns the associations between different objects. By chaining these two modules together, the Local-Global Transformer model reduces computational cost while simultaneously capturing both local and global dependencies in the point cloud, thereby improving the model's learning and representation capabilities. On the other hand, based on the traditional Transformer model, original object geometric structure information is added, and object geometric features such as local plane normal vector, local curvature radius, and normal vector angle are integrated into the self-attention mechanism, thereby designing an efficient Transformer model specifically for processing point cloud data.

[0091] The method disclosed in the present invention does not require a large number of manually designed components, does not require a large amount of prior knowledge, and does not require a large amount of post-processing operations to screen out redundant candidate boxes. The model is simple and can be trained end-to-end, with low computational cost, high processing efficiency and high accuracy.

[0092] The model disclosed in the present invention can be trained end-to-end, and the point cloud data image is annotated, and the bounding box and category of each target are annotated as a training sample. The training sample is input into the point cloud feature extraction network to obtain multiple key points in the output point cloud data, as well as the feature information of each key point; the feature information and position information of each key point are input into the first conversion model, and for each key point, the feature information of the key point is encoded according to the correlation between the key point and other key points within the preset range of the key point to obtain the first feature code of the key point; the first feature code of each key point is input into the classification network, and each key point is classified, and the point classified as the target center is determined as the reference center point; the feature information and position information of each reference center point are input into the second conversion model, and for each reference center point, the feature information of the key point is encoded according to the correlation between the key point and other key points within the preset range of the key point. The first feature code of each reference center point is encoded based on the correlation between the reference center point and other reference center points to obtain the second feature code of the reference center point. The second feature code of each reference center point is input into the decoder of the second conversion model to obtain the feature vector of each reference center point. The feature vector of each reference center point is input into the object detection network to obtain the position and category of each target in the point cloud data. Based on the difference between the position and category of each target in the obtained point cloud data and the bounding box and category of each labeled target, the point cloud feature extraction network, the first conversion model, the classification network, the second conversion model, and the object detection network are trained. The point cloud feature extraction network and the classification network can be pre-trained.

[0093] The present disclosure also proposes a device for detecting targets in point cloud data. Figure 3 Provide a description.

[0094] Figure 3 The following are structural diagrams of some embodiments of the detection device for targets in point cloud data disclosed in the present invention. Figure 3 As shown, the device 30 of this embodiment includes: a feature extraction module 310 , a first encoding module 320 , a classification module 330 , a second encoding module 340 , and a target detection module 350 .

[0095] The feature extraction module 310 is used to input the point cloud data into the point cloud feature extraction network to obtain multiple key points in the output point cloud data and feature information of each key point.

[0096] The first encoding module 320 is configured to encode the feature information of each key point according to the correlation between the key point and other key points within a preset range of the key point, so as to obtain a first feature code of the key point.

[0097] The classification module 330 is used to classify each key point and determine the point classified as the target center as the reference center point.

[0098] The second encoding module 340 is configured to encode the first feature code of each reference center point according to the correlation between the reference center point and other reference center points to obtain the second feature code of the reference center point.

[0099] The target detection module 350 is used to predict the position and category of each target in the point cloud data based on the second feature code of each reference center point.

[0100] In some embodiments, the first encoding module 320 is used to determine the first feature encoding of each key point based on the self-attention mechanism according to the feature information of the key point, the feature information of other key points within the preset range of the key point, and the relative position relationship between the key point and other key points within the preset range of the key point.

[0101] In some embodiments, the first encoding module 320 is used to input the feature information and position information of each key point, and the feature information and position information of other key points within a preset range of the key point into the first self-attention module of the encoder in the first conversion model; in the first self-attention module, the other key points within the preset range of the key point are used as relative points, and for each relative point, the position information of the key point and the position information of the relative point are input into the first position encoding layer, the second position encoding layer and the third position encoding layer respectively to determine the first relative position encoding, the second relative position encoding and the third relative position encoding of the key point and the relative point; according to the product of the feature information of the relative point and the key matrix and the value matrix in the first self-attention module, the key vector and the value vector of the relative point are determined; according to the product of the feature information of the key point and the query matrix in the first self-attention module, the query vector of the key point is determined; according to the first relative position encoding, the second relative position encoding and the third relative position encoding of the key point and each relative point, the key vector of each relative point, the value vector of each relative point and the query vector of the key point, the first feature encoding of the key point is determined.

[0102] In some embodiments, the first encoding module 320 is used to, for each relative point, use the sum of the first position code of the key point and the relative point and the query vector of the key point as the revised query vector of the key point; use the sum of the second position code of the key point and the relative point and the key vector of the relative point as the revised key vector of the relative point; use the sum of the third position code of the key point and the relative point and the value vector of the relative point as the revised value vector of the relative point; input the product of the revised query vector of the key point and the revised key vector of the relative point and the dimension of the feature information of the key point into the first normalization layer to obtain the weight of the relative point; perform weighted summation on the revised value vectors of each relative point according to the weight of each relative point to obtain the first feature code of the key point.

[0103] In some embodiments, the first position encoding layer, the second position encoding layer and the third position encoding layer are respectively the first feedforward network, the second feedforward network and the third feedforward network, and the first encoding module 320 is used to input the difference between the coordinates of the key point and the coordinates of the relative point into the first feedforward network, the second feedforward network and the third feedforward network respectively.

[0104] In some embodiments, the first encoding module 320 is used to determine the first feature encoding of each key point based on the self-attention mechanism according to the feature information of the key point, the feature information of other key points within the preset range of the key point, the relative positional relationship between the key point and other key points within the preset range of the key point, and the relative geometric structure relationship between the key point and other key points within the preset range of the key point.

[0105] In some embodiments, the first encoding module 320 is used to input the feature information, position information and geometric structure information of each key point, and the feature information, position information and geometric structure information of other key points within the preset range of the key point into the first self-attention module of the encoder in the first conversion model; in the first self-attention module, the other key points within the preset range of the key point are used as relative points, and for each relative point, the position information of the key point and the position information of the relative point are input into the first position encoding layer, the second position encoding layer and the third position encoding layer respectively to determine the first relative position code, the second relative position code and the third relative position code of the key point and the relative point; The geometric structure information of the key point and the geometric structure information of the relative point are input into the geometric structure encoding layer to determine the relative geometric structure weight of the key point and the relative point; the key vector and value vector of the relative point are determined according to the product of the feature information of the relative point and the key matrix and the value matrix in the first self-attention module respectively; the query vector of the key point is determined according to the product of the feature information of the key point and the query matrix in the first self-attention module; the first feature code of the key point is determined according to the first relative position code, second relative position code, third relative position code and relative geometric structure weight of the key point and each relative point, the key vector of each relative point, the value vector of each relative point and the query vector of the key point.

[0106] In some embodiments, the first encoding module 320 is used to, for each relative point, use the sum of the first position code of the key point and the relative point and the query vector of the key point as the revised query vector of the key point; use the sum of the second position code of the key point and the relative point and the key vector of the relative point as the revised key vector of the relative point; use the sum of the third position code of the key point and the relative point and the value vector of the relative point as the revised value vector of the relative point; input the product of the revised query vector of the key point and the revised key vector of the relative point, the relative geometric structure weight of the key point and the relative point, and the dimension of the feature information of the key point into the first normalization layer to obtain the weight of the relative point; perform weighted summation on the revised value vectors of each relative point according to the weight of each relative point to obtain the first feature code of the key point.

[0107] In some embodiments, the first encoding module 320 is used to divide the product of the corrected query vector of the key point and the corrected key vector of the relative point by the square root of the dimension of the feature information of the key point, and then superimpose it with the relative geometric structure weight of the key point and the relative point. The result is input into the first normalization layer to obtain the weight of the relative point.

[0108] In some embodiments, the geometric structure information includes: the normal vector of the local plane where the key point is located, at least one of the radius of curvature of the local plane where the key point is located, and the first encoding module 320 is used to determine the relative geometric structure weight of the key point and the relative point based on at least one of the distance between the key point and the relative point, the dot product of the normal vector of the local plane where the key point is located and the normal vector of the local plane where the relative point is located, the difference between the radius of curvature of the local plane where the key point is located and the radius of curvature of the local plane where the relative point is located, and the angle between the normal vector of the local plane where the key point is located and the normal vector of the local plane where the relative point is located.

[0109] In some embodiments, the relative geometric structure weight of the key point and the relative point decreases as the distance between the key point and the relative point increases; the relative geometric structure weight of the key point and the relative point increases as the dot product of the normal vector of the local plane where the key point is located and the normal vector of the local plane where the relative point is located increases; the relative geometric structure weight of the key point and the relative point increases as the difference between the curvature radius of the local plane where the key point is located and the curvature radius of the local plane where the relative point is located increases; the relative geometric structure weight of the key point and the relative point increases as the angle between the normal vector of the local plane where the key point is located and the normal vector of the local plane where the relative point is located increases.

[0110] In some embodiments, the second encoding module 340 is used to determine the second feature coding of each reference center point based on the self-attention mechanism according to the first feature coding of the reference center point, the first feature coding of other reference center points, and the relative position relationship between the reference center point and other reference center points.

[0111] In some embodiments, the second encoding module 340 is used to input the first feature code and position information of each reference center point, and the first feature code and position information of other reference center points into the second self-attention module of the encoder in the second conversion model; in the second self-attention module, the other reference center points are used as relative center points, and for each relative center point, the position information of the reference center point and the position information of the relative center point are input into the fourth position coding layer, the fifth position coding layer and the sixth position coding layer respectively to determine the fourth relative position code, the fifth relative position code and the sixth relative position code of the reference center point and the relative center point; according to the product of the feature information of the relative center point and the key matrix and the value matrix in the second self-attention module, the key vector and the value vector of the relative center point are determined; according to the product of the feature information of the reference center point and the query matrix in the second self-attention module, the query vector of the reference center point is determined; according to the fourth relative position code, the fifth relative position code and the sixth relative position code of the reference center point and each relative center point, the key vector of each relative center point, the value vector of each relative center point and the query vector of the reference center point, the second feature code of the reference center point is determined.

[0112] In some embodiments, the second encoding module 340 is used to, for each relative center point, use the sum of the fourth position code of the reference center point and the relative center point and the query vector of the reference center point as the revised query vector of the reference center point; use the sum of the fifth position code of the reference center point and the relative center point and the key vector of the relative center point as the revised key vector of the relative center point; use the sum of the sixth position code of the reference center point and the relative center point and the value vector of the relative center point as the revised value vector of the relative center point; input the product of the revised query vector of the reference center point and the revised key vector of the relative center point and the dimension of the first feature code of the reference center point into the second normalization layer to obtain the weight of the relative center point; and perform weighted summation on the revised value vectors of each relative center point according to the weight of each relative center point to obtain the second feature code of the reference center point.

[0113] In some embodiments, the fourth relative position encoding, the fifth relative position encoding and the sixth relative position encoding are respectively the fourth feedforward network, the fifth feedforward network and the sixth feedforward network, and the second encoding module 340 is used to input the difference between the coordinates of the reference center point and the coordinates of the relative center point into the fourth feedforward network, the fifth feedforward network and the sixth feedforward network respectively.

[0114] In some embodiments, the classification module 330 is used to input the first feature code of each key point into the classification network to obtain a classification result of the key point; and determine whether the key point is the point of the target center based on the classification result.

[0115] In some embodiments, the classification network is trained using the position information of each key point with labeled information as training data, wherein, for each key point, when the key point is within the bounding box of a target and is the point closest to the center of the target, the labeled information of the key point is the point of the center of the target.

[0116] In some embodiments, the target detection module 350 is used to predict the position and category of each target in the point cloud data based on the second feature code of each reference center point, including: inputting the second feature code of each reference center point into the decoder in the second conversion model to obtain the feature vector of each reference center point; inputting the feature vector of each reference center point into the target detection network to obtain the position and category of each target in the point cloud data.

[0117] In some embodiments, for each key point, other key points within the preset range of the key point are determined using the following method: for each key point, other key points are sorted from small to large according to their distance from the key point, and a preset number of other key points are selected in order from front to back as other key points within the preset range of the key point.

[0118] The detection device of the target in the point cloud data in the embodiment of the present disclosure can be implemented by various computing devices or computer systems. Figure 4 as well as Figure 5 Provide a description.

[0119] Figure 4 The following are structural diagrams of some embodiments of the detection device for targets in point cloud data disclosed in the present invention. Figure 4 As shown, the device 40 of this embodiment includes: a memory 410 and a processor 420 coupled to the memory 410, and the processor 420 is configured to execute the method for detecting targets in point cloud data in any of some embodiments of the present disclosure based on instructions stored in the memory 410.

[0120] The memory 410 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, a database, and other programs.

[0121] Figure 5 FIG. 1 is a structural diagram of another embodiment of the device for detecting targets in point cloud data disclosed in the present invention. Figure 5As shown, the device 50 of this embodiment includes: a memory 510 and a processor 520, which are similar to the memory 410 and the processor 420, respectively. It may also include an input / output interface 530, a network interface 540, a storage interface 550, etc. These interfaces 530, 540, 550 and the memory 510 and the processor 520 can be connected, for example, via a bus 560. Among them, the input / output interface 530 provides a connection interface for input / output devices such as a display, mouse, keyboard, and touch screen. The network interface 540 provides a connection interface for various networked devices, such as a database server or a cloud storage server. The storage interface 550 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0122] The present disclosure also provides an article sorting device, Figure 6 Provide a description.

[0123] like Figure 6 As shown, the item sorting device 6 includes: a detection device 30 / 40 / 50 for targets in point cloud data in any of the aforementioned embodiments, and a sorting component 62 for sorting items corresponding to the targets according to the positions and categories of the respective targets in the point cloud data output by the detection device 30 / 40 / 50 for targets in the point cloud data.

[0124] In some embodiments, the device 6 further includes: a point cloud acquisition component 64 for acquiring point cloud data of a preset area and sending the point cloud data to the detection device 30 / 40 / 50 of the target in the point cloud data.

[0125] The sorting component is, for example, a robotic arm, and the point cloud acquisition component is, for example, a 3D camera.

[0126] The three-dimensional point cloud target detection technology proposed in this disclosure can be applied to products such as vision-based sorting robotic arms in logistics scenarios. That is, the point cloud data collected by the three-dimensional camera mounted on the sorting robotic arm can accurately locate and identify each item, thereby helping the robotic arm to sort them one by one.

[0127] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0128] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0129] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0131] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A method for detecting an object in point cloud data, comprising: Inputting the point cloud data into a point cloud feature extraction network to obtain a plurality of key points in the output point cloud data and feature information of each key point; For each key point, the feature information of the key point is encoded based on the feature information of the key point, the feature information of other key points within a preset range of the key point, and the relative positional relationship between the key point and the other key points within the preset range of the key point to obtain a first feature code for the key point; Classify each key point and determine the point classified as the target center as the reference center point; For each reference center point, encode the first feature code of the reference center point according to the first feature code of the reference center point, the first feature codes of other reference center points, and the relative positional relationship between the reference center point and the other reference center points to obtain a second feature code of the reference center point; The position and category of each target in the point cloud data are predicted based on the second feature code of each reference center point.

2. The detection method according to claim 1, wherein The encoding of the feature information of each key point according to the feature information of the key point, the feature information of other key points within a preset range of the key point, and the relative positional relationship between the key point and the other key points within the preset range of the key point includes: For each key point, the first feature code of the key point is determined based on the self-attention mechanism according to the feature information of the key point, the feature information of other key points within the preset range of the key point, and the relative position relationship between the key point and other key points within the preset range of the key point.

3. The detection method according to claim 2, wherein The method of determining, for each key point, the first feature code of the key point based on the self-attention mechanism according to the feature information of the key point, the feature information of other key points within a preset range of the key point, and the relative positional relationship between the key point and the other key points within the preset range of the key point includes: For each key point, input the feature information and position information of the key point, and the feature information and position information of other key points within a preset range of the key point into the first self-attention module of the encoder in the first conversion model; In the first self-attention module, other key points within a preset range of the key point are used as relative points. For each relative point, the position information of the key point and the position information of the relative point are input into the first position encoding layer, the second position encoding layer, and the third position encoding layer respectively to determine the first relative position code, the second relative position code, and the third relative position code of the key point and the relative point; Determine the key vector and value vector of the relative point according to the product of the feature information of the relative point and the key matrix and value matrix in the first self-attention module; Determine a query vector for the key point based on the product of the feature information of the key point and the query matrix in the first self-attention module; The first feature code of the key point is determined based on the first relative position code, the second relative position code and the third relative position code of the key point and each relative point, the key vector of each relative point, the value vector of each relative point and the query vector of the key point.

4. The detection method according to claim 3, wherein The determining of the first feature code of the key point according to the first relative position code, the second relative position code, and the third relative position code of the key point and each relative point, the key vector of each relative point, the value vector of each relative point, and the query vector of the key point includes: For each relative point, the sum of the key point, the first position code of the relative point and the query vector of the key point is used as the modified query vector of the key point; The sum of the second position codes of the key point and the relative point and the key vector of the relative point is used as the revised key vector of the relative point; The sum of the third position codes of the key point and the relative point and the value vector of the relative point is used as the corrected value vector of the relative point; The product of the modified query vector of the key point and the modified key vector of the relative point and the dimension of the feature information of the key point are input into the first normalization layer to obtain the weight of the relative point; The correction value vectors of each relative point are weighted and summed according to the weight of each relative point to obtain the first feature code of the key point.

5. The detection method according to claim 3, wherein The first position encoding layer, the second position encoding layer, and the third position encoding layer are respectively a first feedforward network, a second feedforward network, and a third feedforward network, and inputting the position information of the key point and the position information of the relative point into the first position encoding layer, the second position encoding layer, and the third position encoding layer respectively comprises: The difference between the coordinates of the key point and the coordinates of the relative point is input into the first feedforward network, the second feedforward network, and the third feedforward network respectively.

6. The detection method according to claim 1, wherein The encoding of the feature information of each key point according to the feature information of the key point, the feature information of other key points within a preset range of the key point, and the relative positional relationship between the key point and the other key points within the preset range of the key point includes: For each key point, the first feature code of the key point is determined based on the self-attention mechanism according to the feature information of the key point, the feature information of other key points within the preset range of the key point, the relative position relationship between the key point and other key points within the preset range of the key point, and the relative geometric structure relationship between the key point and other key points within the preset range of the key point.

7. The detection method according to claim 6, wherein For each key point, determining the first feature code of the key point based on the self-attention mechanism according to the feature information of the key point, the feature information of other key points within a preset range of the key point, the relative positional relationship between the key point and the other key points within the preset range of the key point, and the relative geometric structure relationship between the key point and the other key points within the preset range of the key point includes: For each key point, input the feature information, position information, and geometric structure information of the key point, and the feature information, position information, and geometric structure information of other key points within a preset range of the key point into the first self-attention module of the encoder in the first conversion model; In the first self-attention module, other key points within a preset range of the key point are used as relative points. For each relative point, the position information of the key point and the position information of the relative point are input into the first position encoding layer, the second position encoding layer, and the third position encoding layer respectively to determine the first relative position code, the second relative position code, and the third relative position code of the key point and the relative point; Inputting the geometric structure information of the key point and the geometric structure information of the relative point into the geometric structure coding layer to determine the relative geometric structure weights of the key point and the relative point; Determine the key vector and value vector of the relative point according to the product of the feature information of the relative point and the key matrix and value matrix in the first self-attention module; Determine a query vector for the key point based on the product of the feature information of the key point and the query matrix in the first self-attention module; The first feature code of the key point is determined based on the first relative position code, the second relative position code, the third relative position code and the relative geometric structure weight of the key point and each relative point, the key vector of each relative point, the value vector of each relative point and the query vector of the key point.

8. The detection method according to claim 7, wherein Determining the first feature code of the key point according to the first relative position code, the second relative position code, the third relative position code and the relative geometric structure weight of the key point and each relative point, the key vector of each relative point, the value vector of each relative point and the query vector of the key point includes: For each relative point, the sum of the key point, the first position code of the relative point and the query vector of the key point is used as the modified query vector of the key point; The sum of the second position codes of the key point and the relative point and the key vector of the relative point is used as the revised key vector of the relative point; The sum of the third position codes of the key point and the relative point and the value vector of the relative point is used as the corrected value vector of the relative point; The product of the modified query vector of the key point and the modified key vector of the relative point, the relative geometric weight of the key point and the relative point, and the dimension of the feature information of the key point are input into the first normalization layer to obtain the weight of the relative point; The correction value vectors of each relative point are weighted and summed according to the weight of each relative point to obtain the first feature code of the key point.

9. The detection method according to claim 8, wherein The product of the modified query vector of the key point and the modified key vector of the relative point, the relative geometric structure weight of the key point and the relative point, and the dimension of the feature information of the key point are input into the first normalization layer to obtain the weight of the relative point. The product of the modified query vector of the key point and the modified key vector of the relative point is divided by the square root of the dimension of the feature information of the key point, and then superimposed with the relative geometric structure weight of the key point and the relative point. The result is input into the first normalization layer to obtain the weight of the relative point.

10. The detection method according to claim 7, wherein The geometric structure information includes: at least one of a normal vector of the local plane and a curvature radius of the local plane. The determining of the relative geometric structure weight between the key point and the relative point includes: The relative geometric structure weight of the key point and the relative point is determined based on at least one of the distance between the key point and the relative point, the dot product of the normal vector of the local plane where the key point is located and the normal vector of the local plane where the relative point is located, the difference between the curvature radius of the local plane where the key point is located and the curvature radius of the local plane where the relative point is located, and the angle between the normal vector of the local plane where the key point is located and the normal vector of the local plane where the relative point is located.

11. The detection method according to claim 10, wherein: The relative geometric structure weight between the key point and the relative point decreases as the distance between the key point and the relative point increases; The relative geometric structure weight of the key point and the relative point increases as the dot product of the normal vector of the local plane where the key point is located and the normal vector of the local plane where the relative point is located increases; The relative geometric structure weight of the key point and the relative point increases as the difference between the curvature radius of the local plane where the key point is located and the curvature radius of the local plane where the relative point is located increases; The relative geometric structure weight of the key point and the relative point increases as the angle between the normal vector of the local plane where the key point is located and the normal vector of the local plane where the relative point is located increases after passing through the feature propagation layer.

12. The detection method according to claim 1, wherein The encoding of the first feature code of each reference center point according to the first feature code of the reference center point, the first feature codes of other reference center points, and the relative positional relationship between the reference center point and the other reference center points includes: For each reference center point, the second feature code of the reference center point is determined based on the self-attention mechanism according to the first feature code of the reference center point, the first feature codes of other reference center points, and the relative position relationship between the reference center point and other reference center points.

13. The detection method according to claim 12, wherein: The method of determining, for each reference center point, the second feature code of the reference center point based on the self-attention mechanism according to the first feature code of the reference center point, the first feature codes of other reference center points, and the relative positional relationship between the reference center point and the other reference center points includes: For each reference center point, input the first feature code and position information of the reference center point and the first feature code and position information of other reference center points into the second self-attention module of the encoder in the second conversion model; In the second self-attention module, other reference center points are used as relative center points. For each relative center point, the position information of the reference center point and the position information of the relative center point are input into the fourth position encoding layer, the fifth position encoding layer, and the sixth position encoding layer, respectively, to determine the fourth relative position code, the fifth relative position code, and the sixth relative position code of the reference center point and the relative center point. Determine a key vector and a value vector of the relative center point according to the product of the feature information of the relative center point and the key matrix and the value matrix in the second self-attention module; Determine a query vector for the reference center point based on the product of the feature information of the reference center point and the query matrix in the second self-attention module; The second feature code of the reference center point is determined based on the fourth relative position code, the fifth relative position code and the sixth relative position code of the reference center point and each relative center point, the key vector of each relative center point, the value vector of each relative center point and the query vector of the reference center point.

14. The detection method according to claim 13, wherein The determining, based on the fourth relative position code, the fifth relative position code, and the sixth relative position code of the reference center point and each relative center point, the key vector of each relative center point, the value vector of each relative center point, and the query vector of the reference center point, comprises: For each relative center point, the sum of the reference center point, the fourth position code of the relative center point, and the query vector of the reference center point is used as the modified query vector of the reference center point; The sum of the fifth position codes of the reference center point and the relative center point and the key vector of the relative center point is used as a revised key vector of the relative center point; The sum of the sixth position codes of the reference center point and the relative center point and the value vector of the relative center point is used as the correction value vector of the relative center point; Inputting the product of the modified query vector of the reference center point and the modified key vector of the relative center point and the dimension of the first feature code of the reference center point into the second normalization layer to obtain the weight of the relative center point; The correction value vectors of the respective relative center points are weightedly summed according to the weights of the respective relative center points to obtain the second feature code of the reference center point.

15. The detection method according to claim 13, wherein The fourth relative position encoding, the fifth relative position encoding, and the sixth relative position encoding are respectively a fourth feedforward network, a fifth feedforward network, and a sixth feedforward network, and inputting the position information of the reference center point and the position information of the relative center point into the fourth position encoding layer, the fifth position encoding layer, and the sixth position encoding layer respectively comprises: The differences between the coordinates of the reference center point and the coordinates of the relative center point are input into the fourth feedforward network, the fifth feedforward network and the sixth feedforward network respectively.

16. The detection method according to claim 1, wherein The key points are classified and the points classified as target centers are determined as reference center points, including: For each key point, the first feature code of the key point is input into the classification network to obtain the classification result of the key point; Determine whether the key point is the target center point according to the classification result.

17. The detection method according to claim 16, wherein: The classification network is trained using the position information of each key point with labeled information as training data, wherein for each key point, when the key point is within the bounding box of a target and is the point closest to the center of the target, the labeled information of the key point is the point of the target center.

18. The detection method according to claim 1, wherein The predicting the position and category of each target in the point cloud data according to the second feature code of each reference center point includes: Inputting the second feature code of each reference center point into the decoder of the second conversion model to obtain the feature vector of each reference center point; The feature vectors of each reference center point are input into the target detection network to obtain the position and category of each target in the point cloud data.

19. The detection method according to claim 1, wherein For each key point, other key points within the preset range of the key point are determined using the following method: For each key point, other key points are sorted from small to large according to their distance from the key point, and a preset number of other key points are selected in descending order as other key points within the preset range of the key point.

20. A device for detecting an object in point cloud data, comprising: A feature extraction module is used to input point cloud data into a point cloud feature extraction network to obtain a plurality of key points in the output point cloud data and feature information of each key point; a first encoding module, configured to encode, for each key point, the feature information of the key point based on the feature information of the key point, the feature information of other key points within a preset range of the key point, and the relative positional relationship between the key point and the other key points within the preset range of the key point, to obtain a first feature code for the key point; The classification module is used to classify each key point and determine the point classified as the target center as the reference center point; a second encoding module, configured to encode, for each reference center point, the first feature code of the reference center point, the first feature codes of other reference center points, and the relative positional relationship between the reference center point and the other reference center points, to obtain a second feature code of the reference center point; The target detection module is used to predict the position and category of each target in the point cloud data based on the second feature code of each reference center point.

21. A device for detecting an object in point cloud data, comprising: processor; as well as A memory coupled to the processor, for storing instructions, wherein when the instructions are executed by the processor, the processor executes the method for detecting an object in point cloud data according to any one of claims 1 to 19.

22. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 19 are implemented.

23. An article sorting device, comprising: The detection device and sorting component for targets in point cloud data according to claim 20 or 21; The sorting component is used to sort the targets according to the position and category of each target in the point cloud data output by the detection device of the target in the point cloud data.

24. The article sorting device according to claim 23, further comprising: The point cloud acquisition component is used to collect point cloud data of a preset area and send the point cloud data to a detection device for the target in the point cloud data.

Images